DOCOflSlT BESOBE 

ED 177 324 CI .>22 826 

AUTHOP DoxlaAd, Kathleen A. 

TIx'.c; Vocational Education Outcomes: An £vc.iuative 

bibliography cf ^ni piri'-i^ 1 Studiei,. EibJio<jrai-hy 
series No. 49. 

INvSTii nrxLh Ohio Stat€ Oniv., Columbus. National Center lor 

Fof^oarch it Vcca'i;ionaI Education. 
::"PONi: AuiiNcif Bureau of Occupational and Adult Education (JDH£«/Oii:) , 

Washington, D.c. 
tUB LATL 79 
CUNIFAJI 300-78-0032 

iiO*£. 135p. ; For rtlattd documents Fee C£ 0^2 6iJ-B2D a&a 

ED 170 560 

cDko iftlCii KF01/PCi)6 Plus Postage. 

DiiSwftjL Pluhb Abstracts; Annotated Biblioqraphies ; Educational 

uLjectives; *Ivaluaticn Criteria; £vdluation Method^;; 
lostsecondary Educaticn; Prograni Develotment ; Progran. 
Ktfectiveness ; *Prograiii Evaluation; Research; 
Hf:5€aich Pevievis (Publications) ; Secccdary tducation; 
*Vocational Education 

A£i part of a project designed tc identity appropriate 
and rt:a3iblfc mothcds of pvalua*ing vocational education with respect 
to ita outcomes, this biblioQraphy contains -descriptive and 
eva-iuatiVfc annotations of a selection of empirical studies of 
vocdtioaai education outcomes. Following a technical intrcduccion 
UxscuSwing the methodological issues emphasi2€d in the evaluative 
comoitntti, uiirty-one descriptive annotations are given. Ine 
annotation cf each of the studies presents a discussion of xhe 
rt'Sfarch questions and hypotheses; th€ study design, sample, and tim*. 
fraoit; the statistical analysis; and the results and conclusions. It 
concluUtfs with aa evaluative segment addressing seme cf the 
aethouologicdl and substantive issues inherent in each study. A 
summary epilogue provides a discussion or the findircs and 
mtthodo^ogical isoue? in the collection cf studies. Three inaextf 
(author, iifOLsor, and educat i cr.al level) arc provided. (I.HA) 



* roUut: Mon:-; f^^uppli<>d ly Fnr5 iif th*^ Lest th^l cha i;*.- :r.id< v 

* 1 roiP th* Tiqir. JocuB.ent, ♦ 



ERIC 



Bibliography Series No. 49 



VOCATIONAL EDUCATION OUTCOMES! 
AN EVALUATIVE BIBLIOGRAPHY 
OF EMPIRICAL STUDIES 

% 

Kathleen A, Bolland 



The National Center for Research 
in Vocational Education 
Tho Ohio State University 
1960 Kenny Road 
Columbus, Ohio 4 3210 . 



1979 



US Of I^AtTMEKTOr MiAlTM 
COUCATtON 4 Wil^AttC 
NATlOMAt IKSTlTUTf Qp 
COUCArtON 



••f *'l «M»N .« -^M .AN /A» ON .Mj.(..v 
A* M . t p., Nf. (M ^ ow oi. *, r>Ms 

C\ / • * A. SA» ..SA. N..» ♦ », 

^ * .■ A* . S 1.. . , p,,^ ^ 



THE NATIONAL CENTER MISSION STATEMENT 



The National Center for Research In Vocational Education's 
mission is to increase the ability of diverse agencies, 
institutions, and organizations to solve educational problems 
relating to individual career planning, preparation, and 
progression. The National Center fulfills its mission bys 

• Generating knowledge through research 

• Developing educational programs and products 

• Evaluating individual program needs and outcomes 

• Installing educational programs and products 

• Operating information systeims and services 

• Conducting leadership development, and training 
programs 



) • 



Report of a Project Conducted 
Under Contract No. OE 300780032 



The material in this publication was prepared pursuant to a 
contract with the Bureau of Occupational and Adult Education, 
Office of Education, U.S. Department of Health, Education, and 
Welfare. Contractors undertaking such projects under government 
sponsorship are encouraged to freely express their judgment in 
professional and technical matters. Points of view or opinions 
do not, therefore, necessarily represent official Bureau of 
Occupational and Adult Education, U.S. Office of Education 
position or policy. 

U.S. DEPARTMENT OP HEALTH, EDUCATION, AND WELFARE 



Office of Education 



f 

■rAHi.i: or rorrriiNTS 

Page 

^ ;*;.;V(^KI) ; ^ _ V 

I. l\*TRC)ni'(Tl)\ I 

.i. TfXMlNICAl NOTKS ON ANNOTATINr, THH 

•^MMPK^Al. MrT('<lMK SI'DDIKH 3 

• :;Mi'iKuV\i, oc'raxMi; sTn)ii;s 22 

A. r.itU of Tit. U\s 22 

! . Ci^atiouH and Annotatifnis 25 

:* • :piii.c;M;: HfU7u>iTf'i.AT](.:; A/^h a::^;kssmknt 105 

* • U6 

-\* A;;th':r ijuU'X » 116 

. . Spom^of Index 121 

liiiuc.j t iona'l- 1 li^.<l(^x 125 

LIST ul' TABlJiiS 

* Page 

i. '^fu'cklisf c't c'omponcMits of 

• i:. i/.M 1 ua t ion R;»r)(>ft. 7 

;i. SfApu* t*omnion Roso.irch Dosicfns 12 

:;!. ha t a-c;()I lo(?t ion M^.^thculs 14 

i' * i :»*v>;:>» > (^:tf»n Not (Mfarly Oet'lnod — 

R'':ujtt.ant inansworod Qin^^tions 107 

• Mf»t bMi (M 1 CHu»r.t i^MU'> Htianswc^rfHl 

1 r: Many nutrorno Studi<\^^ j08 

i I I 



FOREWORD 



Vpcational education has been evaluated through meny 
empirical studies in recent years, partly in response to the 
evaluation mandate included in the Vocational Education 
Amendments of 1968. Although context, inputs, and processes 
are still subject to scrutiny, iacareasing attention has been 
focused on products, short-term §ffects, and long-term 
impacts as the concept of accountability has beer appUed to 
all areas of education^ 

This bibliography provides descriptive annotations of 
thirty-one studios, evaluative comments on each of those 
studios, a technical introduction discussing the method- 
ological issues emphasized in the evaluative comments, and 
a summary epilogue discussing the substantive findings and 
methodological issues in the collection of studies reviewed. 
Thus, this document may be useful to a variety of audiences — 
from vocational education specialists to professors and 
graduate students to employment and training planners, 
.Congressional staff, and other individuals interested in 
vocational education. 
^> 

• Vocational Education Outcomes: An Evaluative Bibliog- 
raphy of Empirical Studies originally appeared under thf^ 
title 'Vocational Education OuLcouie Studies: An Evaluative 
Annotated Bibliography Including an Appendix of Key Related 
Literature," Attachment A in Vocational Educatior Outcomes , 
Final Report on Year One of the R&D Project "Examining 
Vocational Education Outcomes and Their Correlates," prepared 
by Robert I,. Darcy, Kathleen A. Bolland, and- Joanne Farley, 
with the assistance of Carolyn M. Taylor, and submitted to 
the U.S. Office of Education in January 1979. The appendix, 
published as a separate document, is titled Vocational Educa- 
tion Outcomes: Annotated Bibliography of Related Literature . 
The report included tvo other components, both published as 
separate documents: Vocational Education Outcomes: 
Perspective for Evaluation ; and Vocational Education Outcomes* 
A Thesaurus of Outcome Questions T Further information on 
these publications is given on the inside back cover of 
this document. 

rn the development of the general report and ' s various 
components, a number of people made valuable contrioutions . 
Diittinquishod reprosontati ves from business, labor, and 
education who served on the National Workshop Panel on 
Vocational Education Outcomes wcro: Walter G. Davis, /^mori- 
.-an Federation of Labor and Congress of Industrial Organiza- 



tions; Mary Ellen Hillaire, Evergieen State College (Washing- 
ton); Addison S. Hobbs, Michigan Department of Education 
(now with the Maryland Department of Education) ; O. Louise 
Lothspeich, Oregon State Advisory Council^ on Vocational 
Education; Fred D. Mack, Central State University (Oklahoma); 
Gerald Q. Miller, Sharon Steel Corporation (Pennsylvania) 
and Employers National Job Servi-ce Improvement Committee; 
Philip h. Smith, The Ohio State University; Jerry C. Olson, 
Pittsburgh Public Schools (Pennsylvania); Wallis E. Pereira, 
Industry Education Council of California; Clio S. Reinwald, 
Arizona Department bf Education; Annell L. Simcoe, Rutgers 
University; and J. Robert Warmbrod, The Ohio State University. 

We are also grateful to Michael D. Hock of Worthingtou, 
Ohio, Douglas Sjogren of Colorado State University, and 
Michael Scriven of the University of San Francisco, who served 
as consultants for the study; and to the five members of the 
National Center's Evaluation Technical Advisory Panel — 
Carol B. Aslanian, College Entrance Examination Board; George 
C, Copa, University of Minnesota; Donald W. Drewes, Conserva 
Inc.; Ruth P. Hughes, Iowa State University; and Daniel L. 
Stuff lebcam. Western Michigan University — .!or their ideas 
and encouragement. 

On behalf of the National Center, I want to express 
appreciation to the Bureau of Occupational and Adult Educa- 
tion, U.S. Office of Education, for sponsoring this study, 
to Robert L. Darcy of the National Center staff for direct- 
ing the Vocational Edu . ition Outcomes project, and to 
Kathleen A. Bolland for preparing this bibliography. 

Robert E. Taylor 
Executive Director 
The Nat^pnal Center for Research 
in Vocational Education 
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VOCATIONAL EDUCATION OUTCOMES: AN EVALUATIVE 
BIBLIOGRAPHY OF EMr-iRICAL STUDIES^ 

I. INTRODUCTION 

This bibliography is the product of a review of voca- 
tional education outcomes literature, with a focus on 
empirical studies. The review was conducted at the outset 
of a proiect designed to identify appropriate and feasible 
methods for evaluating vocational education with respect to 
its outcomer; and the bibliography was compiled as a resource 
document for project staff, the vocational education commun- 
ity, and others interested in examining vocational education 
outcomes. Cpntaining descriptive and evaluative annotations 
of a selection of empirical studies of vocational education 
outcomes, it ^thus provides a sampling of both substantive 
findings and methodological issues. 

Vocational education outcomes are broadly defined as 
the consequences of vocational programs. These consequences 
can be at^tributed to a student's participation in a voca- 
tional program and/or the existence of ongoing vocational 
programs in the community. Outcomes may affect an individual 
student, society as a whole, or some segment of society such 



^This bibliography was prepared as nart of an applied, 
research and development project of the i^Iational Center for 
Research in Vocational Education, "Examining Vocational 
Education Outcomes and Their Correlates," under a contract 
with the Bureau of Occupational and Adult Education, Office 
of Education, U.S. Department of Health, Eaucation, and 
Welfare. Other products of this study include an essay on 
ev^iluating vocational education with respect to outcomes, 
a thesaurus of vocational education outcome questions, and 
an annotated bibliography of literature related to vocation- 
al education outcomes. Further information on these pro- 
ducts is provided on the. inside back cover of this document. 
The c'thor wishes to express appreciation to George Copa, 
Douglas Sjogron, and Daniel L. Stufflebeam for their review 
of an earlier draft; to Trfjha Arthur for her assistance in 
qa-hering documents for review, to Jeanette McConaughy for 
editorial assistance, to Pauline Jacobs and Pam Davis for 
creative secretarial support; and her special thanks to Robert 
L. Darcy, Joanne Farley, Carolyn M. Taylor, and other col- 
ieaquos at the National Center for their helpfu. comments and 
suqqot-Motis concerning this evaluative bibliography. 
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as taxpayers, en^loyers, or a specific community. Vocational 
programs may be at the secondary or postsecondary level; they 
may include programs for adults and out-of-school youth. 
All results, consequences, impacts, or effects of vocational 
education are considered 'outcomes whether, positive or nega- 
tive, intended or unintended, short-term or long-term, 
economic or noneconomic. 

Empirical studies are defined as those studies in 
which data are collected and analyzed in a way that can be 
verified" and/or replicated by other researchers. Thus, the 
bibliography includes such documents as reports of local 
follow-up studies of former vocational students, annual 
avaluation reports prepared by state vocational education 
advisory councils, and doctoral dissertations addressing . 
specific vocational education outcomes. Excluded from the 
bibliography but included in a companion volume^ are general 
data-base reports; review and synthesis papers; evaluation 
methodology papers, evaluation studies focusing on student 
characteristics, context, resources, goals, or processes of 
vocational education; and other publications that are 
related to vocational education evaluation. 

The studies annotated in this bibliography were selected 
through a two-stage process of identification and review. 
In the first stage, a variety of sources was consulted in a 
search for documents published since 1968 addressing topics 
under the broad heading of vocational education evaluation. 
This encompassing identifier was used in the initial stage 
since the more narrow term "vocational education outcome 
studies" is infrequently used and does not even elicit 
immediate recognition from many vocational education research 
era. During this first-sta^e search, staff of the National 
Center and other vocational education professionals ware 
•jonnultod, two computer searches wore initiated,^ and bibli- 
o<jraphios of already-collected documents were screened. 



*^Carolyn M. Taylor, Robert L. Darcy, and Kathleen A. 
Bo 1 1 and , Vocational Education Outcomes; Annotated Bibliogra 
phy of Key Related Literature (Columbus: National Center 
for Rr:soarch in Vocational Education, The Ohio State Univer- 
sity, 10 79^ . 

''A search of the Educational Resources Information 
Center (HRIC) system was conducted and references were 
retrieved and scanned for relevance to ^ho project. Among 
the (loser iptors used were vocational education, vocational 
toUow-up, and proqiam evaluation. A search of the Compro- 
h<»nsive Dissertation Index was conducted and references 



In the second staqo the idontifiod documents were re- 
viewed. Those not reporting omtiirical studios of the 
consequences of vocational *educ<ition were excluded from the 
bibliography. Those retained for the bibliography wore 
selected to: 

« 

1. Reflect the broad scope of vocational education 
outcomes research 

2, Illustrate evaluation research and analysis 
methods as well as findings 

}, Identify research constraints and related problems 
4, Include primarry sources insofar as possible 

-« 

Using this idcinti f ication and review process, thirty- 
one studios wore seloctod, filbiiog raphies of those documents 
wore reviewed for other studies and National Center, staff and 
outside consultants were asked to identify other studies 
that should bo included. Although several other studies 
exist, these jiourcos indicated that they were not so different 
from those already identified as to constitute important 
adfiitions. Thus/ this listing does not exhaust the available 
literature, nor is it a random sample of the qualitative 
variation in research ac vity. Basically, however, it is 
inHicativo uf both the substantive and methodological range 
of evaluations of vocational education outcomes^ published 
since 196B, 



•"retrieved and scanned. Among the descriptors used were 
vorat ional , postsecondary , secondary-, adult, dropout, outcome, 
rol low-up, rndi<in, handicapped, disadvantaged, and bilingual, 
In.bc'^ searches most of the documents identified were not 
report?- of empirical outcome studios, 

'^SiiK'O completidh of the bibliography, however, several 
studios that would have merited inclusion have been brought 
to the author's attention. A review of these studies and 
(if tho procedures followed for selectinci studies in general 
rr-vt-aloci two primary reasons for their exclusion. Some of the 
(lo<Mjnu'ntn wore not identified at all during the selection pro- 
i-t':-;.--; , cither because of a mismat^rh between the I^RIC descrip- 
torn assiqnod to the document and tho5^e used in the search 
or because they were not in the KRIC system and were not 
identified throuffh another source. Other documents were 
i.i«'tit i t led but re jtK'tod on the basis of their title<=: and so 
w»«re not rt'vii'wod. Vor example, thi^ title A Longitudinal 

.y* '''.'V*^.]i.\'l;l.L !2V.yilJl?Il"l*:.f'L :^IL:'_ .L'»'"gr<^m KvaTuation : 
I a-^ l i}jt\ii .t oj. lu.Q~ij^u] um\p' 1 .^^^ nTj g i V; r \'o( • at.Ton a~l G uT d a n c ( ^ 
by ,iei-.,rT,e k.ipes, Thonfas k\ litv ir-rlo i n ViTur'Ra'ncla 1 1 B.' Mar- 
Mi; 1 1:-, i ".'i >; i r 7 Park; Tfie Penruv/ 1 " i n i .i jU.ite f'niversity, 1974) 




The annotation of each of the thirty-one studies begins 
with a discussion of the research questions and. hypotheses; 
the study design, sample, and time frame; the statistical 
analyses; and the results and conclusions. It concludes with 
an evaluative segment addressing some of the salient method- 
ological and substantive issues inherent in each study. Al- 
though applied researchers, including evaluators, must 
balance methodological concerns with concerns for factors 
affecting the utility of results (such as timeliness) , the 
latter issues are more difficult to assess- through reading* 
an evaluation report. The evaluative comments, thus, are 
intended not to serve as meta-evaluations^ but to alert the 
reader to factors influencing the credibility of the reported 
results. Therefore, questions are raised concerning such 
issues as appropriateness of design, generallzability of 
findings, and clarity of reporting. Notable features and 
strengths of the study are also highlighted. The technical 
notes following this section provide a further discussion of 
some methodological issues underlying the evaluative comments 

ED numbers, where provided in the citations, refer to 
the accession number sequentially assigned to documents as 
they are processed into the ERIC systems and published in 
the monthly abstract journal. Resources in Education . (RIE> ^ 
published by the National Institute of Education) . The \ 
documents, except as noted in RIE, are available frcm-ERiC 
Document. Repi )duction Service in either microfiche or paper 
copy. The address is: P.O. Box 190, Arlington, Virginia 
22210. Also, many university libraries have collections 
of ERIC microfiche. When no ED number is given to. a partic- 
ular document, it presumably has not been entered in the 
ERIC system. « ^ 



'^ju'jgests a study of how programs were developed and 
how evaluation information was used in curriculum planning 
and vocational guidance, and so the document was not 
reviewed. 

^>.ota-nvaluation, that is an in-depth critical evalua- 
tion — by a third party — of an evaluation study, is becoming 
increasingly popular. A thorough meta-evaluation requires 
access to full contextual information regarding the study 
and so cannot be done simply through reviewing an evaluation 
roport. For further discussion, see Daniel !•. Stuff lobeam, j 
Meta-evaluation , Occasional Paper Series, No. 3. Kalamazoo: 
The Evaluation Center, Western Michiqan University, 1974). 
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II. TECHNICAL NOTES ON ANNOTATING 
THE EMPIRICAL OUTCOME STUDIES 

^ 

. Evaluation is a term defined in several ways, depending 
on thd orientation of the user. A standard educational 
i^evaluation text*> lists five different usages of the term. . 
For this bibMography , I did not choose any one meaning, but 
rather annotated documents that are characterized as evalua- 
tion reports by their .authors or that are tised as evaluation 
reports oy some audience. However, I maintain that regard- 
loss of definition, the process- of evaluation is a process* 
of inquiry. Although evaluation as a generic term may differ 
from res earch , ' when evaluat'ors set out to inquire about a 
program they are not exempt from the. reujonis of research, from 
the scientific tenets of "the conduct of inquiry. "8 
Although evaluatora^.must balance concern for methodological 
issues such as validity with concern for practical issues • 
iucb as timeliness, it is my contention that a timelV eval- 
uation report of questionable validity is a more serious 
error than an untimely but valid report. The latter report 
jnay bo of use to an audience other t.- r-u that for which it 
was intended, whereas the former is of little use to anyone." 
Thus, although empirical research is not the only leqitirhate • 
evaluation strategy, and a case may be made for a multipli- 
city of strat -riies (including the case study approach, for 
instance), the evaluative comments are made f'rom an empirical 
research perspective. ' j • 

D<\scrl'p tivG and Evaluative Annotations ^ ' 

A. checklist of important r'omporionts of an evaluation 
report was developed^ and used to organise each descriptive 



^ ^Blaine R. Worthen and Jamc^-s R. Sanders, Educational 
Eva^.uatio n : Theory and Practice . ( Wo r thing ton , Ohio: ' 
Charles A. Jones Puolishing Co., 1973) p. 20. 

7 

For a discussion of this issue, see Worthen and Sanders, 
Education al Evaluation , pp. 26-38. 

A short bibliography of research design texts and 
ova luation report checklists will conclude these technical 
notos. Sec, for instance, Abraham Kaplan, The Conduct of 
Ingulrv; Mf^thodolocTy for B ehavioral {science. (Scran tonT 
F\nit>. r~ cTiandler, 1964) . " — 

'*Thosc> c'ompononts wore not -drawn from any ono scnjroe; 
h(>W( vor, distnisHicms of thoni can ho found in sovoral ros(>.u-rh 
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annotation (see Table I). Its sii^iilarity to one that would 
characterize any other scientific research report is evident. 

The evaluative cbmmehts are. more judgmental than the 
descriptive annotations » both in their substance and in the 
selection of issues on which to express professional judgment 
Although the comments address the methodological issues 
surrounding the components of the evaluation report checklist 
space limitations prevent detailed critiques of every issue. l 
The evaluative. comments focus on selected aspects of credi- 
bility and policy utility, as far as can be determined from 
the informatibn given in the report: (1) To what extent 
are the conclusions warranted? (2) Are the conclusions 
relevant to the stated purpose of the study? and (3) Can 
the conclusions be generalized in such a way that they 
contribute to a body of knowledge about the effects of 
vocational education? Suggestions for iir^roving further 
research are made, and the completeness and clarity of the 
reporting are addressed as well. 

To provide a basis for the assessment of credibility 
and utility, cer:.tai^ information must be presented in the 
report and that information must indicate that certain meth- 
odological practices were followed. The evaluative comments 
focus on some salient issues. For example, what aspects of 
the report contribute to the reader's credence in the con- 
clusions? What was done expecially well, and what was 
lacking? It cannot be overly stressed that every issue is 
not addressed in every annotation. 

Following "is a brief discussion of the issues that are 
emphasized. It should be understood that although an i^deal 
study would be methodologically perfect, situational con- 
straints and imperatives (lack of adequate time, money. 



^design texts such as those listed in the reference 
section concluding these notes. Terminology and relative 
• emphasis vary among authors (reflecting diversity in usage 
i by reseajpfchers) , but there is little disagreement about 
' the basic componenti^. 

^^A thorough critique of an evaluation report could • 
easily take five to ten pages, and some are even longer. 
See, for instance, the critiques published in The Journal- of 
Vocational Education Research 1:1 (Winter 1976). 
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expertise, . . .) can prevent that goal from: being reached; 
these constraints and imperatives are not always discussed 
in an. evaluation rep<5rt. The negative comments therefore 
are not meant as .'C-rogations but are intended to alert the 
reader to factors attenuating the credibility of the con- 
clusions. 



Table I 

CHECKLIST OK COMPONENTS OF AN EVALUATION REPORT 
Shatomont of |vur{<)5?o, hypotheses, questions 
Connextuai factors 
Population, sample >■ 
DoBLcjn, data procedures, instruments 
Data treatment and analysis 
Results 

Conclusiona* implications, recommendations 

Sfionaor of study, professional identity of evaluator 



Complctonoss and Clarity of Reports 

It is important that sufficient detail be included in 
the report to allow readers to mentally reconstruct the 
procedures followed, the instruments used, and the groups 
studied. The readers need this information to judge the 
appropriateness of the methodology and to determine how 
similar the programs and students studied are to those they 
are comparing. Thus, not only must all the requisite infor- 
mation be given, it must be provided without ambiguity. 
Terms such as "program completers" and "early leavers" must 
be defined, differences between such groups as "vocational 
students" and "nonvocational students" must be explained, 
and the exact data collection and analysis procedures must 
be described. 

Multiple reports are sometimes written, each with 
information important to. a specific audience. In such cases, 
onch version may not contain all the information discussed 
hero. However, when multipto reports have been written. 



each report should make reference to the other reports, in- 
dicating the audiences to which they are addressed and the 
scope of the Included information. 

When inquiry is conducted for exploratory purposes, 
detail sufficient to enable the readers to grasp the general 
idea of what was done is adequate. However, when the purpose 
of research is to test an hypothesis, the amount of detail 
must be such that researchers who read the report can rep- 
licate the procedures if they wish. (The topic of verifica- 
tion is covered more fully in the next section.) The 
accumulation of a body of knowledge is dependent on repli- 
cation of research. Although statistical analyses can 
indicate the likelihood that an observed phenomenon is a 
function of chance, repeated replications of procedures 
yieldiiig the same findings are necessary to provide further 
assurance that they were not chance occurrences. 

• 

Statement' of Purpose, Hypotheses, Questions 

When engaging in inquiry, one must distinguish between 
the "logic of discovery" and the "logic of justification." 
The logic of discovery applies to the formulation of hypoth- 
eses. There are several methods of formulating hypotheses: 
for example, they may be based- on theory, they may be ground- 
ed in observation, or they may be strictly hunches. There 
are no formal rules that exhaust the possible procedures for 
discovery of hypotheses, but in both scientific and everyday 
discourse, a number of formal procedures for confirming or 
verifying hypotheses can be identif ied. •'■'^ These procedures 
fall under the rubric of the logic of justification; some of 
them will be discussed in the following sections. Because 
tho suitability of methodological procedures depends on the 
purpose of the study (discovery or justification), it is 
important that an evaluation report include a purpose state- 
ment, 

Tho specific objectives, which should be stated as 
evaluation hypotheses or questions, as well as the overall 
purpose, will have implications for the choice of methodology. 



^^Some philosophers of science suggest that an hypothesis 
derived from a particular theory is more acceptable than one 
derived from no particular susbstantive context. See» for 
example, Karl R. Popper, The Logic of Scientific Discovery . 
Rev. ed. (London: Hutchinson and Co., 1968), Chapter 30. 

^^see, for example, Kaplan, The Conduct of Inquiry . 
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Again, in order for the reader to judge the suitability of 
the methodology, the specific objectives of the evaluation 
endeavor must be stated. The reader is cautioned to review 
the methodology used in every study to see if it is appropri- 
ate and adequate to answer the questions posed. 



Contextual Factors 

Anv contextual factors that may affect the interoreta- 
tion of the results should be discussed. The time frame is 
an important example: When was the study conducted? How 
much time elapsed between the occurrence and the study of it? 
When wore the data analyzed and reported? These time factors 
are important because, for example: (1) Some event may have 
occurred during the study affecting its results, even though 
not directly related to vocational education, (2) One may 
question the accuracy with which respondents recall details 
of their first jobs, if questioned ten years later, (3) The 
accuracy of the researcher's recollections is questionable 
when data are analyzed two or three years after collection. 
Other contextual factors include geography and events that 
are not necessarily indicated by time-frame information 
(events with far-reaching consequences are generally known 
by their dates, but more local events are usually, recalled 
from a different standpoint) . * 



Population and Sample 

The study's population is the universe of people, ob- 
jects, or events in which the evaluator is interested. 
Statistically significant results of a scientific study can 
bo generalized to the designated population if appropriate 
sampling techniques are used (treated further in the Data 
Analysis section) , 

In some cases, researchers are able to collect data 
from or concerning the entire population. In most cases 
this is not feasible, however, so the researcher selects a 
sample from the populatidn. Appropriate sampling techniques 
must be used to ensure that results from the sample are rep- 
resentative of results that would be obtained if the whole 
population ..were studied. The sampling procedure used, % 
therefore, must be specified. Sampling techniques and issues 
aro described in various survey research texts as well as in 
, some more yonciral research texts. 

When a sample is chosen but some data are missing be- 
caiU5e not everyone (a) was available for an interview, (b) 
roturnod the questionnaire, or (c) answered all questions, 
til" ,s'il)sot atrt.ually '. -^ed in the study is sometimes referred 
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to as the study sample. What constitutes the study sample 
and the bases on which it differs from the original sample 
should be specified. 

It is important that full information be reported 
concerning the population, the original sample, and the study 
sample. Terms must be defined, procedures must be described 
(how the sample was chosen, what rules Were used to determine 
which data were unusable . . .), and numbers must be provided 
(size of the population, size of the original sample, size 
of the study sample). For example, if it is merely stated 
that the response rate was 64 percent i it is not clear whether 
the study sample was 64 percent of the population or of the 
original sample. Conclusions reached on the basis of a large 
percentage of returns from a small but carefully selected 
sample may be more valid than those based on a small percent- 
age of returns from a large population even if the population 
and the study saunple sizes are equal in both cases. 

Bias is an important issue related to sampling. When 
the original sample^and/or study sample are not representa- 
tive of the population, they are said to be biased. Biased 
samples can be purposefully selected, as when a teacher con- 
siders only the best students and then reports that most of 
the students got jobs; or they can occur beyond *the control 
of the researcher, as when students without jobs fail to 
return*' a questionnaire asking about their employment status. 
Not only should researchers choose sampling and data-gather- 
ing techniques to minimize bias, but they should report 
relevant characteristics of the study sample in comparison 
with characteristics of the population in an effort to deter- 
mine the existence of bias. For example, in a study where 
sex may make a difference, the ratio of males to females in 
the study sample should be compared with the ratio- of males 
to females in the population. Statistical techniques can be 
used t:o overcome problems of bias in some situations. 

• 

The size of the study sample is one of the factors con- 
tributing to the power of a statistical test. The study 
sample must be large enough to ensure that if there are true 
effects, they can' be statistically demonstrated. This is 
somewhat akin to needing a magnifying instrument powerful 
enough to make observable something that exists but cannot 
be seen with the naked eye. 



^^See, for example, William G. Cochran, Sampling Tech - 
niques (New York: John Wiley and Sons, 19771"^ 
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Design, Data» Procedures, Instruments 

• The choice of research design, procedures, and data- 
collection instruments is influenced by such factors as pur- 
pose, available resources, and political' constraints; trade- 
offs are inevitable. Although the readers of evaluation 
reports are generally not in positions to judge the choice 
of such specifics, they should be able to follow the evalu- 
ator's logic in moving from purpose to design. 

The basic design issue is validity. Does the chosen 
design allow warranted conclusions to be drawn about the 
evalaution questions or hypotheses? One must question 
whether something other than the treatment (in this context 
vocational education or some aspect thereof) can be respon- 
sible for the observed effect. There are many threats to 
validity and several research designs have been developed 
to attenuate them. A brief summary of a few of the common 
research designs, adapted from a clt*ssic text, is presented 
^ in Table II. 

In addition to deciding on a general research design, . 
the evaluator must make several specific decisions relative 
to data, procedures, and instrumentation: (1) Whet is the 
nature of the data to be collected? (2) From whom should 
data be collected? (3) When should data be collected? and 
(4) How, should data be cbllected? The evaluation report 
must indicate the chosen alternatives and their rationale in 
sufficient detail that the reader may judge whether the data, 
procedures, and instrumentation are appropriate and adequate 
to the purpose of the evaluation. These four issues are 
interrelated and complex; only a few of the important factors 
will be discussed here. The research design texts listed at 
the conclusion of these notes provide much greater detail. 
One of them includes a discussion of the limitations of 
research paradigms for evaluation studies. 

The general class of data necessary, to answer the pur- 
pose of an evaluation will be more or less obvious depending 
on the specificity of the purpose statement, and it is the 
Gvaluators who must convince their audiences that they have 
chosen appropriate outcomes to study and appropriate data to 



^Daniel L. Stuff lebeam, Walter J. Foley, William J. 
Gephart, Egon G. Cuba, Robert L. Hananond, Howard 0. Merriman, 
and Malcolm M. Provus, Educational Evaluation and Decis ion 
Making (Itasca, Illinoi.;: Peacock, 1971) , p. 22. 
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Table II 
SOME COMMON RESEARCH DESIGNS 

!• The One-Group Pretest-Posttest Design t One group 
receives a pretest, a treatment, and a posttest* 
There Is no control over the possibility that factors 
other than the treatment were responsible for any 
charges that occurred. 



2. The Static-^Group Compariso n t Two Intact groups are 
chosen. Only one receives a treatment; IrDth re- 
cjplve a pbsttest. There Is no adequate assurance 
th^t both groups were esentlally the same before 
the one group received the treatment. 



3. The Time-Series Experiment : A series of measurements 
is taken on a group, a treatment Is given, and the 
series is continued. Something other than the 
treatment may have caused any observed change in 
pattern. 

4* The N onequlvalfent Control Group Design. : Two intact 
groups receive a pretest, one group receives a treat- 
ment, and both groups receive a posttest. The ex- 
tent to which some of the threats to validity are 
controlled depends on the situation. 



Source: Adapted from Donald T. Campbell and Julian S. 
^ Stanley, Experimental and Quasl-Ex|>erimental 
Designs for Research (Chicago: Rand McNally 
1963) . 
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assess those outcomes. -"-^ Not only must an outcome — for in- 
stance, earnings — be specified, but it must also be opera- 
tionalized; that is, it must be defined in a way that clearly 
indicates how it is to be measured. Further, the evaluator 
must justify the operationalization- «for instance, on what 
basis was a choice made from several possible choices, such 
as weekly earnings, annual earnings, and average earnings. 

The general question of data sources raises- two more 
specific questions: (1) Should the data be collected direct- 
ly from members of the population or from other sources? 
and (2) What comparisons should be made? For example, infor- 
mation about the job duties of a population of employees could 
be sought from a sample of the employees or from their em- 
ployers. Tradeoffs among such factors as accuracy, feasibil- 
ity, and cost are sometimes necessary in choosing a data 
sourer.. The evaluation report should provide a rationale 
for thh choices made. 

If conclusions about the effectiveness of vocational 
education are to be drawn, some comparisons must be made. 
The variables being studied indicate what should b compared, 
e.g., earnings, but for purposes of evaluation, these 
variables must be compared against some standards. The 
standards may be relative, e.g., one group of graduates com- 
pared to another, or they may be absolute, e.g., an employee's 
performance compared to a. stated level of satisfactoriness. 

The choice of a standard or of a comparison group is rela- 
tively easy when there is a theoretical reason to suspect 
that the performance of a group should be different after 
its participation in a vocational education program or 
different from that of a group that did not participate. 
Ideally, the theory would even specify the nature of the 



^^Farther discussions of this point can be found in 
Joanne Farley, Voccitional Education Outcomes; A Thesaurus 
of Outcome Questions (Columbus; National Center for Research 
in Vocational Education, The Ohio State University, 1979) 
and Robert L. Darcy, Vocational Education Outcomes; Per- 
spective for Evaluation (Columbus: National Center for 
Research in Vocational Education, The Ohio s€ate University, 
May 19 79) . 

^^See Eugene J. Webb artd others. Unobtrusive Measures: 
Nonreactive Research in Social Sciences (Chicago : Rand 
McNally, 1966), pp. 5-10, for a discussion of comparisons. 
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difference. When evaluators make coirparisons , thoy should 
specify the rationale for their choice of standards. The 
reader then can judge whether appropriate comparisons have 
been made. When standards are not stated .it should be indi- 
cated that it is left to the readers to compare the descrip- 
tive data presented with whatever standards they prefer. 

How the data are to be collected is ajiother issue that 
involves many tradeoffs. The best technique is often the 
most expensive (see Table III for some data-collection 
methods). Once a meth>a, such as personal interviews or 
mailed questionnaires, has been chosen, an instrument must 
be selected as well. An instrument may be developed for the 
research project or one already available may be chosen. In 
either case, questions of validity and reliability shoud be 
addressed as well as conditions of administering the instru- 



ment. 



Table III 



SOME DATA-COLLECTION METHODS 



METHOD 



STRENGTHS 



WEAKNESSES 



Mf^chanical Devices 
(fi.q. , videotape) 



Elimination of 
human errors 
Reliability 



Hiqh cost 
Complexity 



> jbrtorvation 



Natural settings 
Subtleties per- 
ceived by trained 
observers 



Artificial situation 
created by observer's 
presence 
Variation in reli^ibility 
Meaningful factors pos- 
sibly overlookod by 
observer 



^>uestionnairos Honesty fostered Possibility of low per- 

by anonymity centagc of returns 

low cist iRespondonts* lack of 



understanding 
Possibility of super- 




ficial responses 
Ho>5pon.se rate biased 
I ^>y nature of the 
' qu'">fJt.ion}> 
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Table HI (Continued) 
SOME DATA-COLLECTION METHODS 


METHOD 


STRENGTHS 


. WEAKNESSES 


Interviews 


Provision for 
flexibility 

Provision for 
in- depth, response 


High cost 

Skilled interviewers 

required 
Difficulty of siimmariz- 

i.ng 


Rating Scales 
and Checklists 


Low cost 

Ease of completion 
Objective scoring 
Standardized 
responses 


Not all alternatives 
encompassed 

May tell more about 
respondent than about 
topic under consider- 
ation 


So 1 f ••RcDorts 

•* 


Inclusion of 
emotion-laden 
material ^ 


uirticuity or analysis 
High degree of subjec- 
tivity 
May be threatening 


Tests 


Reliability can 

Items can b^ 

analyzed 
Ease of scoring 


Validity of questions 

may oe oouDurui 
Triviality of questions 
Appropriate alternative 

responses may not be 

provided 

Precision of format may 
mask poor items 


Performance 
Samples 


Direct measure 
of ability and 
mastery 


Difficulty of adminis- 
tering 
High cost 

Problems of reliability 
Artificial situation 


Unobtrusivo 
Moasures (e*g. , 
"woar on carpet**) 

! 


Nonre active 
Not consciously 

biased 
Availability and 

measurability 


Questions of ethics 
Questions of validity 


Sotirco; Adapted frcmt material presented in Worthen and Sanders # 
i Kducational E:valuation# cind some other «ourc^«. 
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It is Important that an instrument be both valid, that 
is measure what it is said. to measure, and reliable, that is 
consistent in its measurement. Neither validity nor relia- 
bility is inherent in an instrument; both depend on factors 
of the situation in which the instrument is used. Still, if 
an instrument has been demonstrated to be valid and reliable 
in a given situa+'ion and it is to be used in a similar 
situation, assertions of probable validity and reliability 
are reasonable. 

There are several kinds of validity^^ and corresponding 
ways to assess them, but for present purposes an example 
will suffice. To assess the capability of Test A to measure 
the abil^-ty of individuals to perform Task X, the test can 
be given to individuals known to have different levels of X 
ability. If the variance in scores on Text A corresponds 
with the known variance in ability to perform X, the test is 
said to have concurrent validity. 

Although there^ ar6 several ways to assess reliability, 
it is a unitary concept. If a measuring device consistently 
yields the same measurement when applied to a phenomenon, it 
is said to be reliable. Thus, a thermometer that always 
reads lOQOC when placed in boiling water is a reliable in- 
strument, whereas one that varies, either randomly or with 
some qther variable, is not. Although the thermometer is 
just as reliable if it always reads 90°C when placed in 
boiiing water, it does not validly measure water temperature.. 
The reliability of tests of knowledge is assessed by compar- 
" ing scores on alternative forms of a test, by comparing 
scores on odd-numbered items with scores on even-numbered 
'items (assuming that overall, all items test the same pool 
of knowledge) , or by comparing scores on two different ad- 
ministrations of the same test. ^ 

Reliability and validity are difficult to distinguish 
and to assess with certain measures of the hypothesized out- 
comes of vocational education. Still, it is important to 
assess them insofar as possible. A questionnaire should not 
be assumed to elicit required information, from the point of 
view of the researcher, without some "reality check." Nor 
should responses to an interview be assumed to be factual. 



17predictive validity, content validity, construct val- 
idity, and face validity are among the several types of val- 
idity that can be assessed. See J. P. Guilford and Benjamin 
Fruchter, Fundamental Statistics in Psychology and Education 
Cth Ed. (New York: McGraw-Hill, 1978). 



16 



Questions may bS misinterpreted in some cases, and in others 
a respondent may not have accurate information, or having it, 
may not want to give it. Questionnaire items may be misun- 
derstood by all individuals in a sample or by some specific 
set of individuals. 

Standard techniques found in research tests for assess- 
ing validity and reliability often cann6t be applied to mea- 
sures of vocational education outcomes because they are 
designed for such measuring devices as multiple-choice tests 
of knowledge, attitude scales, or performance tests. Still, 
it is necessary to determine if the techniques used to gather 
vocational education outcomes' data- are.' valid and reliable. 
A general metHod that can be used in many instances is pilot- 
testing. To pilot-test an instrument or other data-gathering 
technique, evaluators select a small representative sample 
from the population of interest and collect data from that 
sample. They then attempt to verify the information received. 
One procedure is to discuss the data collection device with 
the members of the sample to determine theiif understanding 
of what was being asked. Another is to elicit the same 
information from another source. 'For example, if parents 
are questioned about employment of their children, their 
information can be verified by questioning the children 
themselves and their stated employers. If results of a pilot 
test indicate that the measuring devices accurately measure 
the content of interest to the evaluator and dq-. so consis- 
tently, it is reasonable to assume that the devices are valid 
and reliable for other samples from that population. Anpther 
method that can be used to support claims of validity is the 
use of a panel of subject-matter experts. The panel would 
judge whether the instrument, for example a questionnaire 
or interview schedule, covered the domain of interest. 

Another factor related to data collection should be dis- 
cussed in an evaluation report: the conditions of -instrument 
administration.. These conditions may affect reliability 
and/or validity. For example, little credence can be placed 
in interview data obtained from untrained interviewers or 
from data collected in uncomfortable physical conditions " 
(for example, questionnaires completed in a cold room). 



Data Analysis 

For some purposes, simple presentation of data may be 
sufficient. Even in these cases, thoucih, care should be 
taken to guard against misleading tables. When statistical 
analyses are used, even greater care must be taken. Many 
assumptions and conditions underlie most statistical 
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techniques and although many can be violated without detract- 
ing from the usefulness of a given statistic, some cannot be. ^ 
It is particularly important to meet assumptions when parameters 
such as unemployment rates are being estimated, whereas when 
patterns of outcomes such as job satisfaction arie being ex- 
plored, violations of assumptions may not be so critical. 



Results 

« 

Interpretation of . statistical findings must be done 
with care. ^ Statistical significance indicates that an 
observed effect (for example, higher salaries for vocational 
graduates) is not. likely a function of chance variation in 
the selected sample; but further statistical tests can some- 
times be done to indicate how much of the variation is a 
function of the treatment in question (for example, vocational 
education). With correlations, for -example, when the 
correlation coefficient is squared and then multiplied by 
100, thd resultant percentage indicates the variance explained. 
With large sample sizes, statistical significance is often 
found even though the explained variance is very low; when 
this occurs, it indicates that the results have little prac- 
tical ^significance. Finally, a careful reading of the re- 
port is needed jto determine whether the stated results are 
an outgrowth of the reported data rather than unsubstanti- 
ated statements of -the author's convictions. 



Conclusions, Implications, Recommendations 



Some evaluation reports may simply present data and 
summary results, leaving the policy maker to draw conclu- 
sions. When conclusions are drawn, implications stated, 
and/or recbmmendations made in a research report, they should 
be based on the data. 



IBpor example, underlying many statistical techniques 
are the assumptions that the population is homogeneous and 
that the variables under study arc normally distributed in 
the population. These assumptions are treated in statistical 
texts. 

^^See Darrell Huff, How to Lie with Statistics (New 
York: W. W. Norton and Company Inc. , 1954) . 
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Sponsor of Study » Professional Identity of Evc^luator 

« 

The identity of the sponsor and of the evaluator can 
affect the credibility of an evaluation report, both posi- 
tively and negatively. Some sponsors may be known for re- 
quiring quality work and others for. preferring results that 
support their positions... Similarly, some evaluators are 
recognized for- accurate, unbiased inquiry whereas others- 
become so involved in the subject of their inquiry that 
their results are suspect, 20 



, 20see Ernest Nagel, "The Value-Oriented Bias of Social 
Inquiry," in May Broadbeck (ed. ) , Readings in the Philos'bphy 
of the Social Sciences (New York: The Macmillan Co., 1968). 
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B» Citations and Annotations 



The Adequacy of Vocational and Technical Education , South 
Carolina Advisory Council on Vocational and Techn i ca 1 
Education. Columbia: South Carolina Advisory Council 
on Vocational Education, February 1976, 46 pp. (ED 126 
315) 



DESCRIPTIVE ANNOTATION: 

The South Carolina Advisory Council on Vocational and 
Technical Education culminates its evaluation responsibili- 
ties each year by preparing and publishing an annual evalu- 
ation report. It became apparent to ths members, however, 
that "if one is to adequately evaluate the 'effectiveness' 
of programs, what better perspective exists than that of the 
employers?" This study was initiated to explore 'that often- 
neglected perspective. Many questions were asked: 

• Are technical and/or vocational centers a 
good source of trained, prospective employees? 

• Do the centers meet the needs of business 
and industry? 

o 

• Do they meet the needs of the students? 

• How do graduates of the centers compare 
with other employees? 

*• What is t'he quality of this education? 

Recommendations concerning any aspect of the centers were 
sought as well. 

Ten thousand employers from large companies within the 
state constituted the population for the study; question- 
naires were sent to each of them. Usable returns were 
received by the cutoff date from 1161 employers, a response 
rate of 12%. Characteristics of the respondents are dis- 
cussed and presented in tables. A cross- section of the 
South Carolina business and industrial community and of the 
geographic regions of the state was represented. The 
survey form was mailed in October 1975. The questionnaire 
does not ask when employers had experience with graduates 
of vocat^ional-taand/or technical centers. 
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The survey questionnaire, drawn up by a consulting firm, 
was designed to answer the basic quest io'ns the advisory coun- 
cil had raised. After review by a committee of the council 
and by representatives from vocational and technical educa- 
tion, it was revised. The instrument is 'appended to the studi . 

The data are presented in tabular form; comparisons 
are sometimes made between all respondents and those respon- 
dents who have hired vocational/technical graduates. Some 
data are presented as frequencies and percentages of employers 
giving particular responses to specific questions; others 
are in the form of average responses and rank orders. Some 
of the recommendations are listed, with the percentage of 
respondents making each recommendation. ^. 

Among the conclusions discussed are the following: 



• Employers consider both vocational and 
technical centers a good source of trained 
employees, generally rating those employees 
higher than others. 

• Both vocational and technical centers meet 
the needs of business and industry and cf 
the students, but excel at meeting the needs 
of students. 

• Employers are more familiar with technical 
education centers/colleges than with area 
vocational education centers. 

• Vocational and technical programs are 
oriented more to manufacturing than to other 
categories of business and industry. 



Specific recommendations are made to the State Board 
for Technical and Comprehensive Education, to the State 
Board for Vocational Education, and to the State General 
Assembly. Included are recommendations for continued fund- 
ing and for greater dissemination — to the public — of 
information regarding available programs. 

This study was initiated by the South Carolina Advisory 
Council on Vocational and Technical Education. The Dunn 
and Bradstreet, Inc., Marketing Services Division was hired 
because it had access to the desired population (through 
computer files of addresses) and because the council had 
confidence in its research capabilities. 



EVALUATIVE COMMENTS: 



This study is useful in an exploratory sense because it 
suggests areas for further research. The low response rate 
makes such research necessary before findings from the study 
can be more than suggestive. Additional reseai^ch could focus 
on different occupational areas. For example, do employers of 
trade and industrial students view vocational education differ 
ently from employers of business and office students? 

Another question that can be raised from this study 
concerns the extent of experience employers., have had with 
vocational students and what effect, if an^^that has on 
their perceptions. If employers have hired only two students 
each, one would expect their perceptions to be less indica- 
tive of the results of vocational education than the percep- 
tions of those employing hundreds of vocational students. 

The researchers selected a large sample hoping to 
obviate bias both from respondents with vested interests 
in the program and from those who tend to give artswers they 
believe desired. In further research, an attempt should be 
made to assess response bias,, and techniques of question- 
naire construction should be used to . minimize some kinds of 
bias. For example, a forced-choice format can be used to 
prevent responds nts from giving "desirable" responses. 

One conclusion drawn in this report is that although 
vocational and technical centers meet the needs of business 
an-' industry, they better meet the needs of students. This 
issue may deserve further study. Some questions are 
obvious: 

• What are the policy implications of this 
statement if it is a widely-held eraployer 
perception? 

• wnat are the implications if it is indeed 

true"? 

• Are the implications such that, the issue 
should be studied in depth, i.e., which 
programs meet which needs of which employ- 
ers and which students? 

If further research is warranted, one aspect of that research 
should be a study of student perceptions as compared to 
employer perceptions. 

* * * 
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Animal Health Technicians: A Survey of Program Graduates 
and of Veterinarians . Richatd B, Barsaleau and Henry R. 
Walters. Sacramento: Consumnes River College, March 
1977, 16 pp. (ED 136 891) 



DESCRIPTIVE ANNOTATION: * 

Consumnes River College wished to assess the impact of 
its two-year Animal Health Technician (AHT) program. Two 
studies were conducted to investigate the employment status 
and earnings of AHT program graduates: a survey of the 
graduates-, and a survey of local veterinarians, the employer 
group most likely to hire AHT graduates. A question of 
special interest addressed only to veterinarians asked 
whether graduates of any accredited AHT programs were 
"accepted" by the majority of local veterinarians. 

At the t.'.me of the graduate study, three classes of . 
trainees had I -en graduated and 68 students had received 
A.S. degrees. 'Of these, 47 responded to a mailed question- 
naire (a response rate of 69'*,) : 44 were female*, 34 were 
unmarried, and the age range was 20 to over 35 with 34 
under 25. Although development of the questionnaire is not 
discussed nor is the instrument appended, a list of the topic 
surveyed is provided. Included were employment status, job 
stability, and earnings. The report included the following 
findings: 1) 33 or 70% of the respondents were t .ployed as 
AHTs, 14 or 30% were employed in related fields and one 
was unempxoyed by choice; 2) 15 AHTs out of 22 tallied or 
this issue were spending 25% or less of their time in front- 
office duties, clerical work as opposed to animal care work. 
The data are not statistically analyzed. 

The second survey was mailed to 139 licensed California 
veterinarians (most in the Sacramento Valley) ; 71 responded, 
yielding a' response rate of 51%. Sixty-two of the respon- 
dents employed animal health technicians, of whom 27 were 
graduates of an accredited AHT program, 34 had been trained 
on the job, 1 trained in a high school Regionar^-Occupational 
Proqrain, and 5 trained in proprietary schools. The authors 
state that the practice of on-the-jcb training is diminish- 
ing as more trained AHTs are becoming available. The devel- 
opment of the survey quentionnaire is not described nor is 
Ihv instrumont anpended, but issues covered by the survey are 
1 i s tod . 



Resu:ltis discussed include the following: 

• Thirty-seven responding veterinarians 
indicated a willingness to provide Work 
Experience Stations for AHT students. 

* 

• From 47 to 60 veterinarians checked training 
needs in the areas of identification pf 
animal breeds, laboratory techniques,, 
reception procedures, and hospital safety. 

f Some respondents commented, "All the Consumnes 
people I have met seem quite well informed 
regarding specific practices and their 
modes of operation" and "The only real 
problem I have seen is overconf idence 
which cun lead an AHT to give misinfor- 
mation to a client . . . . " 

The study concludes: "Consumnes River College graduates 
are making their marks and are finding willing employers 
who will use their training and talents in a legal and 
ethical manner." 

The surveys were an in-house project conducted by the 
AHT Program Director, R. C. Barsaleau, D.V.M., and 
H. R. Walters of Career Education/Institutional Research, 
both of Consumnes River College. 

EVALUATIVE COMMENTS: 

The major purpose of this study was to determine whether 
veterinarians in the Sacramento Valley area were accepting 
graduates of accredited AHT programs. As most graduates of 
Consumnes River College's program were employed as AHTs and 
surveyed veterinarians gave a range of positive responses 
concerning the programs, a degree of acceptance can be 
inferred. How much is not clear since issues such as possi- 
ble response bias are not discussed. The study has greatiBr 
potential than was realized, however, or at least than was 
reported: although some responding veterinarians were 
employing program graduates and others employing technicians 
who had received their training in other ways, no compara- 
tive data are reported. 

Because the AHT program is not described, nor is 
development of the questionnaire described or the insts^ument 
appended, results of the study cannot be applied to other 
AHT programs in other areas nor can the study be replicated 
from the information given in the report. 

* * * 
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An AsaesiKent of Benefits Derived from Membership in a 

Vocational Student Organization in the Vodational. Tech - 
nical and Adult Education System ^ Dennis R. Collins. 
Menominee: Center for Vocational* Technical and Adult 
Education, University of Wisconsin— Stout, August 1977, 
92 pp. (ED 145 234) 



* DESCRIPTIVE ANNOTATION: 

The primary purpose of this study was to determine 
whether participation in vocational Student organizations 
(VSOs) results in leadership ability, career growth, 
community involvement, and social development on the part 
of participating students. It was hoped that positive 
findings would lead to greater emphasis on VSOs irKthe total 
spectrum of vocational education curricula. Neithet" specific 
research questions nor hypotheses were stated. J 

Pour populations were identified for this study: 
(1) state directors of vocational education? (2) advisors 
of VSOs; (3) Wisconsin postsecondary VSO members; and 
(4) highly participating Wisconsin VSO members. The first 
two groups were questioned about process and structure 
variables and so will not be considered further in this 
annotation. The VSO member sample consisted of 15% of 
postsecondary members of thirty chapters of VSOs in Wisconsin 
including the Wisconsin Office Education Association, Dis- • 
tributive Education Clubs of America (DECA) , Vocational 
Industrial Clubs of America, VUsconsin Association of Health 
Occupations, and Wisconsin Home Economics Related Student 
Organization. The response rate was 56% of the sample. The 
sample of highly participating VSO members consisted of • 
33 DECA members attending the Wisconsin DECA Junior 
Collegiate Career Development Conference in Madison in 
March 1976. 

Written questionnaires were developed for this study 
and samples are appended to the document. The questionnaires 
were mailed to VSO advisors who were instructed to administer 
them as randomly as possible to 15% of the members. Students 
used a 5-point Likert-type scale to rate the benefits of 
VSOs in terms of certain outgrowths: 

• leadership characteristics — ability to 
speak effectively, to make decisions, to 
exercise creativity 

• career growth--pr ide in work, awareness 
of job qualifications, understanding of 
the effects of career choice on life style 
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• community awareness — involvement in 
political activity and social affairs 
of general interest 

• social development — teamwork participa- 
tion, positive self-image, ability to 
make and keep friends 

The data are presented in frequency /percentage tables 
and in graphs. Data from the highly participating students 
and the randomly selected students are presented side-by- 
side for comparison purposes but no statistical analyses 
of differences have been made. Ratings of four or five 
(1 = none, 3 = moderate, 5 = significant) were considered 
to indicate that benefit was derived in terms of the 
characteristic rated. 

Discussion of , these findings are included: (1) forty- 
eight per cent of the participating members gave two 
leadership characteristics a rating of four or five, and 
a larger percentage gave such ratings to all other leader- 
ship characteristics; and (2) the responses of the randomly 
selected members were not as favorable as those of the 
highly participating members, with ratings of four or five 
being made by only 40-50% of the respondents for most of 
the leadership characteristics. 

The author makes several recommendations for giving 
greater emphasis and support to vocational student organi- 
zations. 

This study was conducted by a graduate assistant at 
the University of Wisconsin — Stout, Center for Vocational, 
Technical, and Adult Education — pursuant to a grant or con- 
tract with the Wisconsin Board of Vocational, Technical, 
and Adult Education. 

EVALUATIVE, COMMENTS : 

/' 

Although the stated purpose of this study was to 
empiricaAly investigate claims regarding th.e benefits of 
vocational student organizations, the data collected toward 
that end were strictly opinions of students. No justifica- 
tion fo/r the assumption that students have the ability to 
assess land report those benefits accurately was given. 
Further, since the researcher- did not randomly select 
members, it is not clear that the respondents to the 
questionnaire constituted a random sample of VSO members. . 
For these reasons alone, it is difficult to come to any 
conclusions about the "actual" benefits of participation in 
VSOs. 
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The rating scale uged is positively biased. It is 
different from most Likert-type scales in that the left 
endpoint is "none" and every other point is positively 
defined, whereas the usual scale is symmetrical with two 
or more negative points, a zero point, and two or more 
positive points. How respondents interpreted this scale is 
not clear. Further, although the author interpreted ratings 
of 4 or 5 to indicate a derived, benefit, laws of probability 
predict that 40% of rejipondents would indicate a benefit 
derived {since, given five choices, random selection would 
result in each choice being selected 20% of the time). . The 
author provides no rationale for his assumption that 
positive ratings by 48% or more of the respondents is dif- 
ferent from chance expectation and for the assumption that 
it indicates VSOs as effective in leadership development. 

The analysis of the data was inadequate. Although the 
author compares responses of randomly selected members and 
highly participating members, he uses no statistical tech- 
jiiques to determine whether differences in responses were 
different from those expected by chance. A statistical 
analysis* such as a chi-square could have been used to 
advantage. In addition; a more serioUs problem exists. It 
is possible .that observed differences between the tWo groups 
are filnctions of "self-selection" rather than of intensity 
of participation in a VSO. It is possible that factors 
causing students to be highly participating also cause them 
to-def/elpp leadership ©haracteristics or to be better able 
to. frhese possibilities are not acknowledged. . 



* * * 



nt>no/Fih-Co st Comparison of Vocational Education Programs . 
St Vtowide Evaluation of Vocational-Technical Education in 
FT orida— Volume T , Marshall A Harris. Tallahassee: | 
Florida State University, Florida State Advisory Council 
on Vo'cational and Technical Education, 1972,. 118 pp. 
(ED 074 223) 

t « 

DESCRIPTIVE ANtsIOTATION: 

The general purpose of this study was to determine 
whether vocational education in Florida has sufficient 
economic payoff to bo considered a favorable medium for 
investing both public and private (student) resources. 
Mothodoloqically, it builds on and improves upon two previous 



cost-effectiveness studies conducted in Florida. Three 
questions are addressed: 

• Do vocational education progra'''>s in 

• ^ Florida have positive benefit-cost 

relationships? 

• Do benefdt-cost relationships betv/een 
programs vary and do -these relationsl?ips 
vary from stpdent t9 student? • . 

• . Can the results of a benefit-cost analysis 

be useful to educational planners and 
decision-makers and to individuals antici- ' - 
pating enrollment in vocational education? 

The population included persons who either ^ completed 
training or left early during the period from August 1968 
through September 1971. The state was divided into four 
geographic regions; and two Area Vocational Centers were 
selected from each of the regions, with inclusion based on 
enrollment and. program factors. When a random sample of 
former students had been selected, questionnaires were 
mailed. Three hundred eighteen usable questionnaires were 
collected from the 820 mailed, a response rate of 39%. 
Responses from engineering students were excluded from the 
study as were those from former students with no training- 
related job experience. A table indicating numbers of 
usable returns and reasons for nonusability by vocational 
programs is provided. 

. The mailed questionnaire asked 16 general questions as 
well 'as specific questions about each job held (title, 
earnings, etc.). Portions were adapted from a questionnaire 
developed and used by other researchers. The questionnaire 
is appended. 

Both public and private costs and benefits are analyzed, 
and technical aspects of the analyses are discussed in the 
body of the report and in appendices. Chi-squareS, Scheff^ 
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J^cob J. Kaufman, Ernst W. Stromsdorfer, Teh-Wei Hu, 
and Maw Lin Lee. A Cost-Effective n ess Study of Vocational 
Education: A Compari son of Vocational and Non- Vor;,f in^^^T- 

"^"15^°"-. ?^?D^J-y^?Pll9J?ls.- Report to the U.S. offi ce 
of Education, Washington, D.C., March 1969. University Park 
Pennsylvania State University, 1969. 



33 

to 



♦ • 



tests, analysis of variance, and multiple regression are 
used to analyze the data. A be*heflt'COst planning model is 
presented as well. 

Among the six "most cogent findings and conclusions" 
discussed* are (1) ". . .on the average, society will recoup 
^its average investment of $1,716 per student in 1.3 years 

* and- on the average, a student will, recoup his average invest- 
*ment of $2,411 in 1.9 years", and (2) -"On both/public and 

student investments, nbnsecondary st idents had] higher rates 
of return than did secondary' studei: .. . " 

EVALUATIVE COMMENTS s 

* This report is notable for being very .detailed. Cost- 
benefit analysis is a complex technique, and detail is a 
necessary prerequisite for accurate assessment*. However, 
there are pitfalls of this type of analysis and findings 
should "be viewed in light of those pitfalls. 23 

A strong point 6€ this study is that a questionnaire in 
a previous study was adapted. If a knowledge base about voca- 
tional education is to be built, some comparability in data 
collection and analysi^s are necessary. r The repeated adminis- 
tration of the same, or at least similar, instruments is one 
method of achievijig this comparability. . 

.t ♦ * * « ♦ 



Comparative Analysis of Post Secondary Occupational and ' 
Educational Outcomes for the High School Class of 1972. 



R. Reid Creech, Norman E. Freeberg, Donald 
Kenneth M. Wilson, and Kan-Hua Young. 



Rock, 

, ^. Princeton: Educa- 
tional Testing Service, May 1977, 465 pp. (ED 139 845) 



DESCRIPTIVE ANNOTATION: 

A 

This report is the result of a project having two 
objectives: "to effect a partial evaluation of the effec- 
tiveness of vocational education as compared to academic 
and general high school programs" and "to develop informa- 
tion useful for program planning specialists involved in 
vocational education." To accomplish these objectives. 



23see Darcy, Perspective for a discussion of some of 
the pitfalls. 
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various aspects of "the current 24 gtatus and activities of 
the class of 1972" were investigated, including educaeional 
activities, job changes, duration of employment, job sAtis- 
factjion, and weekly earnings (Chapter 2) . Also investigated 
was the realization of plans and aspirations" • (Chapter 4) . 
Some chapters of the document describe the social makeup 
of the high school class of 1972, the variables used in the 
study, and the methods of analysis/ others discuss "non- 
response and related concerns" and "Where do we go from 
here?" The major findings of tjhe study are summarized in 
Chapter 5. 

This study is based on data collected in the National 
Longitudinal Study of the High School Class of 1972, one 
of a series "of National Longitudinal Studies of Educational 
Effects sponsored by the Department of Health, Education, 
and Welfare. Based on both the original set of data and on 
the first 18-month follow-up, it "represents the first 
in-depth investigation of whe^t has happened to the Class of 
1972 since its graduation, ^ with special emphasis on educa- 
tional and vocational outcomes." 

Nearly 3,000,000 students graduated from high schools 
across the. nation in 1972. About 18,000 of these students 
were selected, according to a carefully designed and 
executed sampling plan, to participate in the National 
Longitudinal Study of the High School Class of 1972. 

The development of the first follow-up of questionnaires 
(FFQ) , which was used to collect much of the data discussed 
in this report, is not described in this document. A form 
of the questionnaire is appended however and the question- 
naire items are discussed. Nonresponse aild related concerns 
are also discussed. 

The data analysis techniques employed in this study are 
thoroughly described. *'Iultiple point-biserial regression 
analysis was used for comparisons between two groups, and 
multiple discriminant analysis was used for comparisons 
involving three or more groups. An econometric model using 
a simultaneous two-equation system was used to examine the 
interaction between college attendance and work participation. 
Five structural (path) 'analytical models were built and 
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The time period referred to as current roughly repre- 
sents May 1972 - January 1974. 
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tested in an effort to assess what underlies decision making 
in the occupational-educational aspiration area, post-high 
school educational progress. 

* Among the results discussed in the "Summary of Major 
Findings" are these: 

* 

• At the time of the first follow-uo survey, 
64% were employed and 8% were out of work. 

• The employment rates for graduates of vari- 
ous curricula were vocational^; 77%; general; 
68%; and academic; 56%. 

• The homemaking rates- were vocational; 40%; 
general; 36%; and academic; 15%. 

if 

• Earnings were similar for blacks and whites 
but much higher for males than for females. 

• X)n the average, vocational graduates 
worked a greater number of weeks from 
October 1972 to October 1973 than did 
graduates of other curricula. 



The project investigators were from the staff of 
Educational Testing Service. The project was conducted 
pursuant to HEW Office of Planning, Budgeting, and Evalua- 
tion Contract No. 300-75-0312, a response to Office of 
Education RFP 75-60. 

EVALUATIVE • COMMENTS ; 

Time and space do not permit a thorough analysis of 
this document. It is notable In that the variables and 
methods of analysis are described in detail; however, the 
reader is cautioned to thoroughly review both this document 
and other documents describing the National Longitudinal 
Study of the Class of 1972 before accepting any of the 
results listed here. 

* * * 
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A Comparative Study of the Occupational Achievement of 
Vocat^ional and Non-Vocational High School Graduates in 
the State of Arkansas . Dean C. Andrews and Lawrence H. 
Roberts. Magnolia: ^Educational Planning and Evaluation 
Services, December 20, 1974, 131 pp. (ED 112 207) 



DESCRIPTIVE ANNOTATION: 

Thin study was a response to the concerns of taxpayers 
and decision-makers in .regard to the effectiveness of edu- 
cation. Specifically, it was conducted to acquire feedback 
information to be used in improving the vocational education 
programs in Arkansas. A cost-benefit study was considered 
but rejected because, eunong other reasons, the cost would 
have been prohibitive and too many methodological issues 
are yet to be refined. Instead, this study sought to 
compare the occupational achievements of students who were 
enrolled in vocational education with those of students who ' 
were not so enrolled with resr«2ct to their (1) occupational 
status, (2) educational status, (3) earnings, (4) job 
satisfaction, and (5) other, related factors. 

The study, conducted in 1974, had as its pooulahion the 
1970- graduating classes of eight Arkansas public high 
schools. Three samples were selected: (1) those who had 
had two semesters of vocational classes; (2) those who had 
had four semesters of vocational classes; and (3) those who 
had had no vocational training. In order to ensure that 
the third group was comparable to the first two, those 
graduates who had had grade point averages (CPAs) above 2.25 
and had planned to go -to college were eliminated. A total 
of 1749 of the 2597 graduates of tho eight schools were 
sent questionnaires yielding an overall sample of 67%. 
Three mailings were used in an attempt to reach as many 
graduates as possible; the final response rate was 36% of 
the sample (24% of the population) . Tables indicating 
responses to the separate mailings and response rates by 
sex, race, GPA, and vocational training are provided. 

A preliminary questionnaire was developed and pilot- 
tested for clarity and reading level with. eleven 1973 
graduates of one of the high schools used in the study. 
Those graduates made suggestions that were incorporated into 
a revised form subsequently adminis.tered to five more gradu- 
ates. In addition, several professional educators critiqued 
the preliminary questionnaire. A copy of the final form is 
provided in an appendix. The data from school records and 
the questionnaires wore summarized on predesigned tables 
and coded on computer cards. Most of the data analysis is 
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descriptive— summary tabulations of frequencies and percent- . 
ag6s. Chi-square analyses were used to determine whether 
statistically significant differences existed between 
categories. 

Topics discussed . in three chapters include the occu- 
pational activities of the vocational graduates, the 
vocational graduates' evaluation of their training, and 
a comparison of occupational achievement of the four-semester 
vocational graduates and the nonvocational graduates. *A 
subsequent chapter summarizes the results and provides 
conclusions and recommendations. 

, Chi-square* analyses were done to compare the vocational 
and nonvocational groupn on the basis of sexual composition, 
racial composition, and grade point average. The only 
statistically significant difference between the groups was 
in grades: a higheif proportion of vocational graduates had 
CPAs above 2.00. The authors acknowledge that the results 
reported, of which the following are a sample, should be 
interpreted in light of this grade difference: 

■ • Tnere are no signiticant differences in 
occupational status between graduates of 
each group (i.e.» no significajit difference 
in proportions of those working full-time, 
of those attending college* etc.). 

• A greater proportion or vocational graduates 
are employed in trade and industrial areas. 

• There are no significant differences in the 
degree of job satisfaction expressed by the 
vocational and nonvocational graduates. 

• There are no statistically significant 
differences in earnings. 

Based on differences in occupational- status between 
vocational students with two semesters of vocational courses 
and those with fo..r semesters, the authors conclude that 
two years of vocational training is desirable for- those who 
plan to work after high school. Based on the lack of 
relationship between type of job and type of training, 
however, they recommend that local education agencies study 
the relationships of their offerings to the job market and/or 
implement a vigorous placement- program. Still, they conclude, 
vocational graduates have a greater chance than nonvocatiortal 
graduates of finding full-uime work they consider closely 
related to their high school training. 
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This study was conducted by two members of "Educatioiial 
Planning and Evaluation Services/' based in Magnolia, 
Arkansas. 

EVALUATIVE COMMENTS:. . . 

This study is notable for at least "ihree reasons: 

• The authors indicate that they chose a 
methodology in light of their purpose. 

• The data-collection instrument was 
pilot-tested. 

• Oif^rences between groups were 
statistically analyzed. 

Because' graduates who had four semesters of vocational 
training had a higher occupational status than those wxth 
only two semesters, the authors conclude that four semesters 
are better than two. They failed, '.owever, to consider the 
possibility of a self-selection factor. Perhaps those 
students who completed four semesters have other qualities 
helping them achieve a higher occupational status than 
those with only two semesters. There are a number of ways 
this question can be explored in further research. 
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A Comparative Study of the Occupational Achievement of Voca - 
tional and Non-vocational High School Graduates m Texas . 
John A. Laska and Jau-Woei Chiou. Austin: Department of. 
Cultural Foundations of Education, Center for International 
Education, University of Texas at Austin, June 30, 1973, 
183 pp. (ED 118 944) 



DESCRIPTIVE ANNOTATION: 

The primary purpose of this study was "to determine 
•whether there, were any real differences in occupational 
achievement between the high school graduates who had been 
enrolled in vocational education programs and the graduates 
who had not been enrolled in these programs." It was intend- 
ed to respond to the "legitimate right (of concerned and 
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interested taxpayers, legislators, and political leaders) to 
inquire about the efficacy of . . . large expenditures in 
preparing students for gainful employment in the labor market." 
Further, the findings of this empirical research were intended 
to be used fo* continuing improvement of vocational education 
programs . 

The literature review section cites the Little review*-" as 
weir as some specif i-: studies. "In summary, a review of exist- 
ing research indicates that earnings, length of employment,, 
job satisfaction, occupational status, and occupational mobil- 
ity were commonly used as the indices of occupational achieve- 
ment. Some other variables were found relevant in accounting 
for these indices of occupational achievement. They were sex, 
ethnicity, academic achievement , _ socijal origin, college 
attendance, community differences, and vocational education. 
The findings obtained from exis* mg research were extremely 
useful in developing the conceptions and in deciding the 
scope of this study." 

For this study, vocational students were defined as those 
who had completed two credits in a school year in a vocational 
program. Nonvocational students were matched to vocational 
students according to sex and accumulated scholastic ranking. 
Complete information on the sampling procedure is provided. 
The population consisted of students who had graduated in 1970 
from 17 public high schools in 5 independent Texas school dis- 
tricts. A letter and questionnaire were sent to 3,045 gradu- 
ates in August L972. Follow-up mailing was sent in September, 
m all, 1058 (3 5''^.) of the 3, 045 graduates responded to the 
questionnaire, 502. (16%) could not be reached, and 148 5 (49%) 
either indicated a refusal to cooperate or did not return the 
questionnaire by the October deadline. Response rates by 
schools arc tabled as are characteristics of the sample. 

A survey questionnaire was designed to collect data from 
which to derive six indices, of occupational achievements . 
Throe indices concerned income--accumulated income, monthly 
<>arnings, and hourly rate — and three concerned employment — 
weekly^ working hours, months of employment , "and actual job 
length. In order to account for possible spurious, but sta- 
tistically significant, relationships, several control vari- 
ables were introduced, including sex, community, ethnicity, 
social origin, marital status, college attendance, and aca- 
demic achiovomoPt. Other possibly relevant variables, such 



^^J. Kenneth Little. Review and Synthesis of Research on 
the Placement ahd Follow-up of Vocational Education Students 
(Columbus :^ The Center for Vocational and Technical Education, 
The Ohio State University, February 1970). 
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as quality of teaching personnel, teaching facilities, work 
motivation, personality traits, and labor-market situations, 
were deemed beyond the' scope of this study. 

The questionnaire was pilot-tested extensively to ensure 
that it was comprehensible. Item analyses were performed to 
eliminate redundant items. The survey questionnaire is ap- 
pended as are coded raw data«, 

The two major statistical techniques used in this study 
are analysis of variance and multiple regression analysis. 
Factor analysis is used to interpret the interrelationships 
among different indices of occupational achievement. The 
rationale for each specific analysis used is "thoroughly dis- 
cussed and several tables of data are provided. Results of 
preliminary analysis and subsequent analyses are thoroughly 
discussed with accompanying tables. 

Among the conclusions and policy implications (based on 
894 respondents) provided in the final chapter are theses 

• 244 completed two or more years of college, and 157 
attended college from 4 to 18 months during the 27 
months after high school graduation. For those stu- 
dents then working, secondary vocational education 
was not an important variable in initial employment 
when community, sex, and college iattendance were con- 
trolled. 

• The effects of secondary vocational education were 
. statistically significant (.'^5 level or beyond) in 

accounting for the accumulated income and actual job 
length of the 493 noncollege workers of both sexes, 
the months of employment of male noncollege workers, 
and the weekly working hours of female noncollege 
workers in the Austin-San Antonio area, but not in 
Houston.' , 

• Community factors seemed to relate to the operation 
of secondary vocational education in different ways. 

• Beneficial returns of secondary vocational education 
were definitely real and substantive in terms of 
yearly accumulation. 

The researchers are members of the Department of Cultural 
Foundations of Education, Center for International Education, 
University of Texas at Austin. The study was conducted pur- 
suant to a contract with the Department of Occupational Edu- 
cation and Technology of the Texas Education Agency. 
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EVALUATIVE COMMENTS i 



This is a complex study with several notable features. 
Rationale, based in part on a literature review, for the 
chosen variables was provided; the survey instrument was pilo 
tested? the data analyses are thoroughly described; and the . 
raw data are appended to the report. It is one of the few 
studies in which vocational students are compared to nonvoca- 
tional students. The students were matched according to sex 
and scholastic ranking. However, the authors do not acknow- 
ledge the limitations of matching as a tool in quasiexperx- 
mental research. . 

* * * 



The Contributions of .Vocational Education, Training, and Work 
Experience to the Early Career Achievements of Young Men . 
John T. Grasso. Doctoral dissertation presented to the 
Faculty of Educational Development of The Ohio State Uni- 
versity, Columbus, July 1975, 167 pp. (ED 113 537) 



DESCRIPTIVE ANNOTATION: 

The author states "Much of the decision making underly- 
ing vocational education legislation and expenditures occurred 
in the absence of adequate information." He goes on to 
say that although research has been done and some of it is 
encouraging in its diversity of perspectives, the policy rele- 
vance of the research "has been circumscribed by limitations 
inherent in the design of many of the studies, or as a result 
of conceptual or methodological difficulties, often compound- 
ed — if not caused — by a lack of adequate data." Other limi- 
tations are discussed as well. 

This study was conducted to contribute to "improvement 
in the formulation of public policy on the education and 
training of youth ... by reviewing completed research and 
by extending and refining it with both improved data and 
methods of analysis." The empirical aspect of the otudy seeks 
to determine how youth from different high school programs 
comparo in the following areas: 

• knowledge of occupations 

• attitudes toward the adequacy of their 
preparation for work 
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• participation in post-school training 
and learning activities 

« 

• skill level 

• wages earned 

• job satisfaction * 

• unemployment experience 

• long-run career pj^ospects 

Except when noted, the conclusions drawn in this study 
are based on a subset of the National Longitudinal . Surveys 
{NLS) sample: those young men not enrolled in school at the 
time in question who had completed high school but not even 
a year of college. "A national probability sample of over 
5,000 males between the ages. of 14 and 24 was interviewed in 
the last quarter of calendar year 1966 and annually there- 
after through 1971; information collejcted through 1969 was 
available for this study." The young men were divided into 
four curricular categories: college preparatory, general, 
vocational-commercial/ and vocationcil-other (referred to as 
. vocational) . 

This study is based on tHe personal interviews and the 
1968 special mail survey conducted as part of the National 
Longitudinal Surveys. The instrumentation is described in 
various NLS documents. All analyses in the study are based 
on data statistically weighted to allow reliable comparisons 
between black and white youth. Further delimitation of the 
youth on the bases of vocational program studies or extent of 
study were not possible; however, it was possible to statis- 
tically control for the following demographic variables: 
socioeconomic status^^ scholastic aptitude, and race. Other 
control variables used are price level and demand^ military 
service* residence at age 14 (rural farm, rural non-farm, 
small city, medium city, and large city or suburb) , and, for 
the in-school youth analyses, grade placement and work experi 
enct? '{if any) , All the variables used, independent and de- 
pendent, are thoroughly described. 
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The results relative to the eight factors itemized in 
the first paragraph of this annotation are desctibed in de- 
tail with several summary tables* The following are included 
among the findings: 

• Neither vocational nor vocational-commerical 
students possess greater occupational infor- 
mation than general track students. 

• Vocational graduates do not consider them- 
selves better prepared for the world of 

^ * work than general graduates. 

^\ • Many graduates ,pf the various curricula 
\ sought further training — college, pre- 
professional or technical; commercial, 
^ managerial; vocational, skilled manual.. 

• \rhere is no indication that vocational 

graduates obtain jobs with higher skill 
requirements than other graduates, but 
tho^e with post high school training (ex- 
cluding college) obtain jobs with higher 
ratings. 

• In term? ot wages, white vocational students 
benefit more from post high school training 
(excluding college) than other white students. 
However, there were no wage differences found 
among black youth from different curricula. 

• Job satisfaction results* reflect the earn- 
ings results reported in the preceding item 
for white youths, but again no curricular 
differences were found among black youth. 

• No conclusions on unemployment experience 
with respect to curricular differences 
could be drawn from the data. 

• college preparatory students and white commer- 
cial graduates have the most favorable long-run 
career prospects in terms of earnings; however, 
among black youth, the positive differentinl 
for college preparatory students may be non- 
significant. 
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The final chapter presents concisions and policy impli- 
•cations derived from the study. There are at least four 
major implications for secondary school policy maker-s: 

• Schools, need to impart more information jregard*- 
ing careers in general and about the role and 
importance of postsecondary school training and 
learning opportunities. 

• The accessibility of postsecondary training 
programs should be increased. 

• * T • 

• Additional training for black youth must be 
promoted, but efforts must be accompanied by 
affirmative action labor-market policies. 

• Specific types of careers best served by each 
curriculum should be ascertained and programs 
modified accordingly to serve the needs of youth 
and society. 

\ 

* This study was undertaken, in part, to fulfill require- 
ments for the author's doctoral degree. He spent five years 
working on the National Longitudinal Surveys project while 
completing his doctoral studies. 

EVALUATIVE COMMENTS: 

Several limitations of the curripulum variable are acknow 
ledged by the author. Occupational programs, for example, 
auto mechanics and agriculture, are not distinguished; stu- 
dent reports of their curricula were not confirmed? the ex- 
tent or intensity of vocational study undertaken in high 
school was not noted; and students who took a "general" cur- 
riculum with a practical arts emphasis (for example, general 
business cburses) are not distinguished from those who took 
a general curriculum without such emphasis. In spite of its 
.limitations, however, this study with its statistical con- 
trols of several variables is an improvement over several 
less sophisticated studies. 

* 

* * * 
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A Decade of Feedback Information from Harcum Graduateo t 

i966-i9'y'^ » Boris Blai, Jr» Bryn Mawr, Penn.t Harcum 

Junior College, August 1977, 5 pp. (ED 142 270) 



DESCRIPTIVE ANNOTATION t 

The overall purpose of Harcum Junior College's Institu- 
tional Research Reports is to provide information for "effect- 
ing improvements in existing practices." This document is a 
report of their s61f-evaluation program, ongoing since aca- 
demic year 1967-68, designed in consonance with Harvard 
professor David Reisman's statemerft • • « college 
must be tested by its products, the most obvious product 
being the alumni." Specifically addressed is the extent to 
which the College has achieved its objectives "[to] provide 
transfer programs paralleling those offered in the first two 
years of four-year collegiate programs of study," and itoj 
offer career preparation, equipping individuals to become 
contributing members of the working world." 

From 1968 through 1977, a questionnaire was mailed each 
summer to that year's graduates. The average response rate 
was 52%. Although the questionnaire was not discussed, this 
report summarizes the data collected over that decade. 

Among the results presented are these: 

• The lowest percentage of those accepted for 
transfer to advanced standing was 30% in 
1976; the highest, 62% in 1973. 

• The lowest percentage receiving full-time 
employment was 20% in 1973? the highest # 
52% in 19.69. 

« r 

• In response to the statement "... Harcum 
prepared me for the field I have chosen," 
89% responded "very helpful" or "somewhat 
helpful"? only il%, "not helpful." 

Among the interpretive comments made by the author are: 
J'One* reason for the decreasing trend in transfer to four- 
.. year co lieges/universities may be that: more Harcum graduates 
' are entering occupational fields that do not require bacca- 
laureate preparation," and "Some graduates do not look for 
work until the fall following graduation, and therefore the 
percentages reported are conservative estimates of the employ- 
ment status of Harcum graduates." The general conclusion 
of this decade report is "the articulation of its 
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graduates into the community at large . * . (evidences] a 
record of effective results." 

This report was written by the Director of the Harcum 
Junior College Office of Institutional Research. 

EVALUATIVE COMMENTS: . . ' 

This is a very brief report, meant for internal vise at 
the college. As such» it provides policy makers and staff 
with a descriptive overview of some outcomes of HarcUm's 
blend of liberal arts and an occupational program. However, 
the report gives the reader insufficient information for 
judging the credibility of the data. Was the questionnaire 
valid and reliable?. What was the extent of the nonresponse 
bias? 

A further problem is that no standards are provided 
against which to compare the reported percentages. How is 
a finding that 11% of respondents found the college to be 
••not helpful'^ in preparing them for their chosen fields use- 
ful to policy makers? Is that percentage so small as to be 
not worth considering or so large as to challenge the college 
to initiate changes aimed at improving its occupational 
preparation programs? If it is not useful information, one 
may question the inclusion of the question in an ev^aluation 
study* 



* * * 

'» 

The Effectiveness of Vocational and Technical Programs; A 
National Fpllow^p Survey . Gerald G. Somers, in collabor- 
ation with Laurfe M. Sharp and Thelma Myint and with the 
assistance of Susan Pernbach Meives. Madison: Center, 
for Studies in Vocational and Technical Education, Univer- 
sity of Wisconsin, 1971, 282 pp. (ED 055 190) 



DESCRIPTIVE ANNOTATION:. 

The basic purpose of this study was to provide sipmc mea- 
sures of the effectiveness of vocational-technical education, 
that is, formal instruction aimed at preparing youth and/or 
adults for initial entrance into an occupation or foc/ advance- 
ment within an occupation (excluding advancement for which 
baccalaureate or postbaccalaureate education is a prerequisite.) 
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At the time of the study there were no national data 
availa)9le for an evaluation of the effectiveness of vocational 
education; thus. the study was undertaken to fill that void. < 

The universe^ the* sampling techniques » the study samples^ 
and nenresponse bias are thoroughly described and several 
tables -are included. Basically » a stratified random sampling 
technique was, used, resulting 'in samples of graduates from 
high school Vocational programs; graduates from postsecon- 
jdary vocational schools (presumably vocational-technical in- 
, stitutes) ; graduates from vocational programs of junior col- 
leges ;2( graduates of high school academic programs; and 
dropouts from high schools, pr 'secondary schools; and 
.junior colleges. ^ 

% • ^ ♦ 

Samples of 19£6 grhduatles were surveyed by mail in 1969*. 
Questionnaires were mailed to 7 3^^7 graduates of 194 secon- 
dary schools, 3461 graduates of 54 postsecondary schools, 
and 2591 graduates of 64 junior colleges. The samples of 
dropouts obtained we^e too small to include in most of the 
evaluative analyses, but. some descriptive data are provided. 

4 . ' . . * 

A random sample of nonrespondents was drawn and follow- 
up telephone interviews were conducted so nonresponse bias 
could be assessed « There did not appear to be any consis- 
^ tent, significant dif ferences;' howevet, it is noted that the 
^ respondents and sampled* nonrespondents are hot evenly dis- 
tributed among the prbgram areas. . . 

• ^ ^ 

The* development of the mailed questionnaires is not dis- 
cussed; they are appended, however. The questionnaires had 
from 47 to 70 questions, with multiple-choice and open-ended 
items.. The data analysis consisted of cross-tabulafiions of ., 
percentages', chi-squares, and regression^analyses« 

* * 

Findings of the study are discussed in six chapters: 

• The Educational Experience, and Student Attitudes 

• Post-Vocational Employment Experience 

• Post-Vocational Wages and Earnings' 



^^It is not clear whether community .colleges are included, 
but the author cites a report prepared b'' the Bureau of 
Social Science Research giving details on characteristics 
of included junior colleges. 
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• Post-Vocational Educational Experience 

• Evaluation of the Cost§ and Benefits of 
Vocational Education 

• Conclusions and Policy Implications 

Several tables are provided throughout the chapters and 
in an appendix. Seven "highlights" are provided in the front 
matter of the document, and each chapter has a summary con- 
clusion. 

Conclusions and policy implications, in the form of a 
discussion of the significance of school level, geographic 
region, urban vs. rural setting, program area, demographic • 
characteristics, and grade point averages ^s influences on 
attitudes, labor market performance, and further educational 
experience are presented in the final chapter. 

. Several summary observations are included with their 
implicationsj_ * 

5 

• Vocational programs at different school levels 
serve different clientele (because of variances 
in student backgrounds and attitudes) and dif- 
ferent training purposes, despite similarity in 
occupational program titles. 

• A relatively large proportion of junior college 
agricultural, technical, and health program 
graduates and high school distributive and 
trade and industry graduates move directly from 
graduation to a full-time job or further edu- 
cation, whi^reas graduates of other programs 
require longer job-search times. 

• Junior college vocational graduates^*^ general 
experience greater economic succsrfswith more 
satisfactory employment and better earnings 
than: other graduates, both vocational and 
academic . ^ ' 

• With the exception of agricultural programs, 
economic benefits of vocational and technical 
education are higher for graduates in the West, 



The authors acknowledge that not all junior college 
programs are cost-effective; that the cost data used are from 
other studies; and that junior college programs are not neces- 
sarily the best choice for every student because of variance 
in interest, intelligence, and other qualifications. 
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Northeast, and North Central than in ! 
the South; and higher for urban and 
suburban graduates than for rural grad- 
uates. 

• Males » older graduates, and married grad- 
uates enjoyed labor market experiences 
superior to those of females, younger grad- 
uates and unmarried graduates. One impli- 
cation, is that there is need for a longer 
period of general education preceding specific 
vocational choices. 

• The relation of job to training does, not 
appear tb influence levels of employment 
and earnings . Actually many students are 
able to earn more by moving out of their 
field of training upon labor-market . entry . 

• This may support the view that general voca- 
tional training and training for clusters of 
job skills are preferable to specific train-, 
ing and training for specific jobs. 

• Although occupational program area is of 
some importance in the labor market ex- . 
perience of graduates, it is no% as important 

. as traditionally believed. 

This research was funded by the Division of Adult and 
Vocational Research, U. S. Office of Education, Department 
of Health, Education, and Welfare. The senior author is a 
professor at the University of Wisconsin; his collaborators 
are staff of the Bureau of Social Science Research, and the 
junior author is a graduate research assistant at the Uni- 
versity of Wisconsin. 

EVALUATIVE COMMENTS: 

This study is too complex to permit a thorough critique. 
The authors acknowledge some limitations of the data collect- 
ed and the data available, as well as noting that, in many 
instances, data are unavailable to distinguish among alternative 
reasons for some findings. For example, did women as a 
whole earn less because of overt discriminatioit or because 
they did not aspire to full-time jobs following their voca- 
tional training? 

Although the report is lengthy, some information neces- 
sary for assessing the study and relating its findings to 
other studies is lacking. For example, what exactly is a 
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graduate of a pQStsecondary vocational school? What specific 
methods of statistical analysis were used? How was the ques- 
tionnarie developed?. 



* * * 



Employer Satisfaction with the Skills of Vocational Education 
Graduates in North Dakota^ Larry L. Smiley. Grand Porks: 
Bureau of Educational Research and Services, University 
of North Dakota, August 1976, 81 pp. 

DESCRIPTIVF ANNOTATION: 

One of the goals of North Dakota's State Plan' for Voca- 
tional Education specifies the percentage of completers who 
will be placed in the field in. which they have been trained, 
or in a related 'field. This study was done to aid that ef- 
fort by ascertaining how satisfied employers were with the 
completers they had hired and in what areas the vocational 
programs needed to be improved. 

The initial data base comprised all secondary and post- 
socondary siTfes in North Dakota with at least five vocational 
programs. Personnel in each program identified three major 
employers of program graduates and one or two self-employed 
graduates. The .482 names identified constituted the survey 
sample. From 226 employers indicating a willingness to par- 
ticipate in a personal interview, 191 provided data. A • 
questionnaire (appended to the document) was developed for 
use in the personal interview, which took approximately 30 
minutes. 

Twenty-one questions were asked, several with multiple 
parts. Some of the questions were in a rating-scale format, 
some in the form of checklists, and others open-ended or re- 
quiring a yes or no response. They addressed issues of gen- 
eral satisfaction with employees; degree of specific know- 
ledge, skills, and abilities employees possessed; and ways 
of improving training. 

Findings are presented in the order that questions appear 
on the interview schedule. Responses to each question arc 
summarized in tabular form along with a brief discussion. 
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It is left to the reader to draw conclusions'. The reported 
findings include the following: 

• ^ewer than five per cent of the responding 

employers rated the vocational education 
system below average* o 

• 73% of employers rated manual job skills above 
average and 47% ascribed the same rating to 
practical job knowledge. 

• A specific employer comment is "Most of 
the vocational education graduates come to 
work with skil^ls well developed, but the 
training programs have failed to offisr the 
students anything that prepares them for 
moving into supervisory or managerial posi- 
tions." 

." 

Nearly 90% of the employers rated the voca- 
tionally trained employee high to average 
in comparison with those without the voca- 
tional training. Many could not make a com- 
parison because certification requirements 
for entry- level job placement prevented a 
non-vocationally trained individual from 
being hired. 

• Several "needed areas of improvement for 
job applicants" were listed. Heading the 
list (in order of frequency of response) were 
attitudes toward work and appearance. Also 
listed were courses that vocationally trained 
students should have, with courses in Communi- 
cation, Speal^ing Effectively, and Work Orien- 
tation at the head of the list. Need for a 
course in Success in Marriage was listed by 
88% of the respondents. 

• Many employers contact schools to recruit 
qualified employees. Some (including out-of- 
state) schools are regularly contacted because 
such individual schools have gained a reputa- 
tion for quality— good skill development, e-c. 

• A list of some of the general comments employers 
made about vocational education programs and 
the graduates is provided. Among the comments 
arei "Realities of position need to be made 
clear to the students," and "Secretaries trained 
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as medical secretaries are less capable than ''- 
those with no specialized training." 

• Several employers expressed the opinion that 
high school graduates are sold a bill of ggods 
when told they have a saleable skill after their 
high school vocational training, that the intent 
of- high school vocational training is to introduce 
young people to vocational areas and materials 
of those trades, not in any sense to provide a 
terminal degree. One employer said, "If this 
high school level training has helped a young 
person to clarify vocational interest which 
can be furthered by additional schooling, it 
has served an excellent purpose." 



EVALUATIVE COMMENTS: 

This type of study can alert vocational planners to 
employers' perceptions of vocational education, thus indicat 
ing areas where the system is meeting their needs, where 
improvements are needed, and what kinds of changes may prove 
beneficial. It gives direction for more intensive investi- 
gation. A careful reading of the report is necessary, how- 
ever, because in some instances the author notes that re- 
spondents' verbal comments contradict the ratings. For ex- 
ample, employers' ratings of students' attitudes toward work 
do not indicate that this is" a serious problem, whereas em- 
ployers* comments indicate that it is. 

It is unfortunate that the findings are not broken down 
by lev: I of schooling (secondary vs. postsecondary) and that 
information about the content of the various vocational cur- 
ricula was not compiled. Although this study was not intend- 
ed to be generalized to programs outside North Dakota, such 
information would be helpful to those who might wish to use 
th»? study to suggest areas for further exploration within 
their own systems. 



* * * 



A Follow-up study of Former Occupational-Technical Students 
at Virginia Community Colleges." Edith H. Carter. Paper 
presented at the Annual Regional Research Conference for 
Secondary and Post-Secondary Vocational Education Person- 
nel, April 17, 1976,. 10 pp. (ED 136 899) 
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DESCRIPTIVE ANNOTATION? 

It is stated in this report "The planning for expansion 
of program operation should be based upon awareness of in- 
formation about studei:^ts and potential students, their 
occupational needs, and also the activities of former stu- 
dents after they leave the community college.*' This study 
was designed and conducted to provide comprehensive and ac- 
curate information about former students of Virginia commun- 
ity college occupational-technical programs. Explicit, major 
objectives of the study were as follows: 

• To identify selected personal and demographic 
characteristics of former students in occupa- 
tional-technical programs 

• To identify postcollege activities of former 

students ^ 

To study the attitudes of former students 
toward their community college experience 
and current employment 

• To study patterns of student retention and » 
withdrawal 

• To examine differences among graduates and 
nongraduates and among the several types of 
graduates in terms of their characteristics, 
postcoilege activities, and personal evalua- 
tions of college experience and employment 

The population consisted of all students who had been 
enrolled in a Virginia Community College occupational cur- 
riculum at any time from the fall of 1966 through the fall 
of 1969. A total of 11,623 former students from 13 colleges 
were identified. After an initial questionnaire and three 
follow-ups were mailed, 61% of the population (not including 
the 12% undeliverable) returned usable responses (73%\of the 
graduates and 56% of the nongraduates) . \^ 

Two instruments wore used to collect data: The "col lege 
data form" wa,s used to collect demographic information and 
tho "studont questionnaire" was used to collect in^'vrmation 
about postcoilege activities, employment, and evaluation 
of college experiences. The student questionnaire was de- 
signed to fulfill the stated objectives of the study, but 
no further discussion of its development is provided, nor 
IB it appended. 
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Responses to the questionnaire were entered onto computer 
tapes by optical scanning to obviate the possibility of key- 
punch errors. No statistical analyses of the data are re- 
ported. 

Two results reported are (1) that almost three-fourths 
of the (respondents! were work-.ig full-time in jobs related 
to their community college curricula and (2) that both grad"^ 
uates and nongraduates rated the quality of their college ' 
preparation superior or good in most are^s with nine-tenths 
of them ^tating that they would recommend their community ' 
college to an individual planning to enroll in the same pro- 
gram. 

This study was conducted/reported by the Director of the 
Office of Institutional Research, New River Community College. 

EVALUATIVE COMMENTS: 

This report is necessarily brief, because it was a con- 
ference paper rather than a final report of a study. Still, 
the credibility would be enhanced if more complete analyses 
of the data were presented. The relatively large response 
rate reported may be indicative of the advantage of follow- 
up mailings. > 

As in most of the studies annotated in this biSliography , 
thero is some question regarding the validity of the survey 
instrument. For example, what assurance is there that former 
students who respond to a questionnaire from their' alma mater, 
when asked if they would recommend it to other individuals, 
will respond honestly after thoughtful consideration? A 
pilot test in which respondents are questioned more thoroughly 
Ln a personal interview (and asked, for example, if they ever' 
have recommended their college to another individual) is one ■. 
way to assess the validity of the survey instrument. " 



* * * 



A Follow-up Study of Illinois Homo Economics Job Training 
Programs. Final Report . Kathleen M. Howell and Joyce 
L. Folstohauson. Springfield: state of Illinois, Board 
of Vocational Education and Rehabilitation, Division of 
Vocational and Technical Education, Research and Devf>lop- 
ment Unit, September 1971, 122 pp. (ED 059 379) 



DESCRIPTIVE ANNOTATION: 

This pilot study was designed to test the^use of certain 
procedures and instruments for examining the post-high school 
employment experiences of graduates of Illinois Home Econom- 
ics cooperative Education Programs and to provide recommenda- 
tions for leaders in the field. Both graduates and employers 
were surveyed. Questions asked on the graduate questionnaire 
concerned type and number of jobs held, reasons for leaving 
jobs, desirable and undesirable features of jobs, part- or 
full-time stattfs, reasons for never being employed, and the 
value of the job training received. The employer/supervisor 
questionnaire asked how well the graduate, was prepared for 
tho job, how proficient the graduate was in important 30b- 
entry skills, and how tine job training for that position 
could be Improved. Both questionnaires included multiple- 
choice and open-ended questions. 

Cooperative Gainful Home Economics teachers in 44 Illi- 
nois public secondary schools identified 691 home economics 
graduates of 1969 and 1970. The sample analyzed consisted 
Qf 188 respondents, a response rate of 27% (with non-deliv- 
erables assumed to be random, a response rate of 34%) . 
Tho^e graduates identified 110 employers of whom 75 returned 
questionnaires, yielding a response rate of 68%. 

Both the graduate and Lhe employer questionnaires were 
adapted from others developed and used previously by the 
senior author. After pilot testing for use in this study, 
revisions were made. The questionnaires are appended. 

The results were tabulated according to sex, graduation 
year, and courses taken. Tables of frequencies, averages, 
and percentages are provided, but no further statistical 
analyses were made. 

The following findings from the graduate questionnaire 
were included: 

• 62% of the graduate sample were employed at 
the time of the study. 

• 24% to 55% of those employed (varying 
with training area) were employed in 
their areas of training , 



of the graduates had never beon 
omployed . 
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• 51% of, the graduates ind^icated ^ob sat- 
isfaction related to working with people, 
whereas 54% indicated dissatisfaction related 
to job-situation factors, such as specific 
undesirable tasks and working conditions . ' 

• Graduates felt that, although they would 
have liked more employment experiences and 
instruction, their vocational programs had 
prepared them for employment, especially in 
getting along with others, using time and 
energy, and handling new or unpleasant sit- 
uations. 

Following are some of the findings from the employer/ 
supervisor questionnaire reported: 

' • Job training contributed to employability 
traits, especially in areas of attendance, 
attitudes toward supervision, and cooper a - 
tiveness . 

• Personal factors such as initiative and 
honesty were entry-level skills considered 
important. 

• Training should emphasize, positive attitudes 
toward work, learning, ah^ supervision. 

Specific recommendations are made for the Program Appro- 
val and Evaluation Unit, the Special Programs Unit, and the 
Research and Development Unit. These range from recommenda- 
tions that specific steps to implement the provisions of the 
State Plan be identified to suggestions that a comprehensive 
evaluation program include long-range investigations to de- 
termine relationships between crime and delinquency rates 
and job training (or lack thereof) for youth. 

Both authors are from the Eastern Illinois University's 
School of Home Economics. The senior author, project direc- 
tor, is an Associate Professor whereas the junior author, 
principal investigator, is a research assistant. The study 
was conducted pursuant to a contract with the Illinois Board 
of Vocational Education and Rehabilitation. 

EVALUATIVE COMMENTS: 

The authors themselves question the validity of their 
results, since only 34% of the graduates who received ques- 
tionnaires responded. They express concern that 35% of the 
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mailed questionnaires were undeliverable by the post office, 
indicating lack of current address information even though 
follow-up is mandated. 

One of the notable features of this study is its pilot 
test and appended questionnaire. < Reliability and validity 
data are not given but it is noted that some item analysis 
has been done, indicating that the instruments will ,be re- 
fined for still further use. 



» 
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Follow-up Study of Machine Tool Technology and Building Con - 
struction Graduates: Evaluation Report . Wilbur Hall, 
Rodney Gray, and Arthur 0. Berry. South Portland: South- 
ern Maine Vocational Technical Institute, August 1975, 
115 pp. (ED 124 746) 

.[ , 

DESCRIPTIVE ANNOTATION: \ 

This study was des4?gned toNdetermine how well the South- 
ern Maine Vocational Technical institute's Machine Tool Tech- 
nology (MTT) and Building Construotion (BC) programs are suc- 
ceeding in their efforts to prepa)re well-qualified employees. 
Specifically, it sought to acquire the following information: 
(1) employment status of graduates, (2) employment sequence 
of graduates, (3V perceptions of the value of the MTT and BC 
progrt»ms held by graduates and employers, (4) upgrading and 
retraining needs, and (5) graduate, and employer profiles for 
programmatic restructuring and development. 

A questionnaire was sent to each of the 126 MTT and BC^ 
students from the classes of 1970, 1972, and 1974. A randSm 
sample of those graduates living within a reasonable commut- 
ing distance of Southern Maine Vocational Technical Insti- 
tute were selected for personal interviews to supplement the 
questionnaire. Seventy-five graduates returned questionnaires 
and 38 of them were interviewed. Employers were interviewed 
when possible. The response rate was 63% for BC griiduates 
and 55% for 'MTT graduates. A. majority of the sample had re- 
ceivod diplomas; the others, associate degrees. 

Tho questionnaire, consisting of an "opinionnairc" and a 
chocklist, was validated by -use with a sample of graduates 
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before being sent to the selected sample.: Although personal 
information (name and address) .was collected, graduates were 
assured that their responses would be kept confidential. The 
opinionnaire and checklist are appended. 

The document provides extensive tables detailing re- 
sponses to the survey instruments. Responses #0 many of the 
open-ended questions are listed as well. Results, in terms 
of percentages, are reported separately for MTT graduates and 
BC graduates, although no statistical analysis is provided. 

Among twenty summary findings are the following: 

• Ail graduates who responded were employed, 

^ with a majority working in the trade area , 
for which they had been prepared or in one 
closely related to it. 

• The majority had held only one job since . • 
graduation. 

• 38.6% of the BC graduates and SS'. 7% of the 
MTT graduates had .pursued further training 
or education. 

• Respondents indicated general satisfaction 
with the quality of the instruction they had 
received, although attaching many suggestions 
for improvement in the curricula. 

• Employers were generally satisfied with program 
graduates, although they were concerned "with 
continued program relevancy to the trade or 
occupational areas." Recommendations for- pro- 
gram and study improvement are provided. 

This study was initiated by the Southern Maine Vocational 
Technical Institute. The priftcipal investigators were in- 
structors at the Institute — one BC, the other MTT — employed 
full time for two months to conduct the study. Dr. Arthur 0. 
Berry, whose professional identity is not reported, was re- 
sponsible for the data analysis and the evaluation report. 



EVALUATIVE COMMENTS: 

The report gives sufficient information for replicating 
the procedures of the study. However, other information is 
lacking. The interview questions for the graduates and the 
employers are not described; although it is stated that the 
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questionnaire was validated, the 'validation procedure and 
results are not described. Although the 100% employment rate 
of the respoiidents suggests a tendency for bias toward the 
employed, possible bias in the respondents is not consideted. 
And finally, statistical analysis of the data was not done. 

Although /these insufficiencies may have minor impact for 
the intended /audience— Southern Maine may conclude that their 
Vocational Technical Institute is doing a reasonable job of 
preparing v;ell-qu,alif ied' employee? — it is injudicious fo% an 
expanded audience to draw any conclusions from this report, 
or even to , determine whether the survey instrument would he 
useful Jor, another institution. Without even minimal sta- 
tistical data analysis to determine the probability of random 
difforences in responses* on the basis of this survey it is 
risky oven for Southern Maine to draw any substantive con- 
clusions/about policy or program changes needed. 

r ' ' 

♦ * * 



Toliow-up Study of 1969-1975 Graduates of the Division of 
Technology of 'New York City Comnmnity Colleges . Jeanne 
M. Gammel, Stanley M. Brodsky, and Richard L. Alfred. 
Brookfyn: New York City Community College, April 1976, 
134 pp. (ED 121 396) 



DESCRIPTIVE ANNOTATION: 

The preface to this study states, "If the college of the 
future is to realize even a modicum of its potential, faculty 
and administrators will have to begin to pay as much attention 
to the outputs of instructional programs as to the number of 
students, buildings, and organizational structures." The 
study was designed to provide such output information in terms 
of career patterns, perceptions of curricula, and attitudes 
toward work and further education held by graduates of twelve 
curricula of the Division of Engineering Technology at New 
York City Community College (NYCCC) . The results were in- 
tended to bo used not only internally but also externally 
to provide agencies with information regarding cr.itributions 
made by the college to business and industry. 

The 2,554 students graduating over the sevon-year period 
from 1969 through 1975 constituted the population. In the 
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fall of 1975, a questionnaire was mailed to 2,687 of those 
graduates, though the method of selecting the sample is not 
stated (perhaps mailing addresses were not available for the 
rest of the population) . Nine hundred twenty-two graduates 
returned usable responses, yielding an overall response rate 
of 44%. A table lists the number of graduates by curriculum 
and graduation year and also indicates respective response 
rates. 



The development of the questionnaire is not discussed^ 
nor is reliability and validity information provided; however, 
the instrument is appended. The questions asked were designed 
to collect data on the following variables: (1) present occu- 
pational status; (2) field retention of employed graduates; 
(3) proportions in senior, supervisory, -managerial, or pro- 
fessional positions; (4) change of employers and job ti*-les 
between first and present jobs; t5) further college atten- 
dance; (6) further degrees sought or completed; (7) percep- 
tions of the NYCCC curricula; and (8) other variables related 
to employment and. further education. 

The resulting -data for each dependent variable are pre- 
sented in two-dimensional f r;equency/percentage tables of 
graduation year by curriculum. Summary results from four- 
teen earlier follow-up studies are prerented for comparison. 
No statistical analyses were done. The authors point out ' 
that results of this study cannot be considered positive or 
negative because standards for success have not been estab- 
lished. For example, no one has determined how many students 
should be employed in training-related fields or what per- 
centage of students should continue their education. 

The results of the study are discussed in detail and hy- 
potheses are suggested to explain differences in graduates 
of various' years and curricula. Amor\g the results discussed, 
the following are included: 



• 77% of the graduates are employed, 14% are 
full-time students, 9% are unemployed, and 
1% are in the military service or retired. 

• Of the unemployed 1974 and 197 5 graduates, 
most are students, whereas few of the un- 
employed 1969-1971 graduates are students. 

• The proportion of employed graduates by 
curriculum area varies from 47% for arch- 
itectural technology to 96% for firo- 
protection technology. 



• 



• of the 607 graduates employed in training- 
related jobs,* 44% Jiave senior, supervisory, 
managerial, or professional positions. 

# * * * 

• Of all empl-oyed graduate respondents, 27% , 
^ found their' education extremely helpful -n 

performing their present job, 38% fairly 
helpful, 24% slightly helpful, and 11% not. 
at all' helpful. 

• 61% of the respondents attended college 
after graduation from NYCCC, ranging from 
58% for the class of 1974 to 68% for the 
class of 1971. To summarize the findings, 
graduates of each curriculum are described 
in "modal student profiles." • 

Although the study was clearly done internally both by 
and for college staff, the researchers were not identified. 

EVALUATI.VE COMMENTS: - h 

This study was well thought out and is one 9f the few 
in which the particular data collected, ^^^.^^nffault is^L • 
overall purpose. It is quite possible that 
omission in reporting rather than in the study itself : Al- 
though information from fourteen previous studies is pre- 
sented, there is no indication whether the ^ame -instrument 
was U84d. The point is twofold: Ar| the data Comparable 
and has the instrument' been tested? 

Many comparisons are made among graduates of varying cur- 
ricula and graduation years. Although hypotheses were de- 
veloped to Ixplain some of the -differences, no statistical 
Inalyles were performed to indicate whether they were likely 
?o be merely chance differences. Thus, the need for further 
research designed to test those hypotheses i3 indicated. 



* * * 



••A Follow-up Survey of the Graduates of the High Schools m 
Vocational Region <t8. New Hampshire." Roger D. Cnm and 
Eugene W. Ross. Paper presented to the New England Edu- 
ce^ tional Research Organization Annual Meeting, Province- 
town, Mass., May 6, 1976, 32 pp. (ED 123 471) 
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DESCRIPTIVE ANNOTATION t 



This study, conducted to obtain a data base for New 
Hampshire Vocational Region #8 public secondary vocational 
educators, is intended to help them plan future curricula 
and programs. The issues addressed were education and train 
ing beyond high school, employment status during the first 
^ear after graduation, and current employment status. 

The population comprised the 1968, 1970, 1972, and 1974 
graduates of the six public high schools within Region #8. . 
The population was stratified by school and graduating class' 
and a random sample was taken from each of the 24 strata. 
The sample (1266) was 50.5% of the population (2509). 

An interview form was developed, using the Dictionary of 
Occupational Titles as a guide for categorizing vocations. 
The interview schedule is appended. Telephone interviewing 
was conducted by paid, trained interviewers over a two-week 
period. When the high school, graduate could not be reached, 
information was collected, when possible, from a parent, ' 
grandparent, guardian, or spouse. Information regarding 840 
graduates was collected, yielding a response rate of 66.4%. 
The response rates for all strata are tabled. 

The data were tabulated in frequencies and percentages, 
and a composite profile for the region was obtained., No ' 
further .statistical analysis was done. 

Among the 36 findings listed are the following: 

• 48% of the giiaduates received no additional 
. education or training. 

• 80% of the graduates were employed immediately 
after graduatipn. 

• At the time of the study, 63% of the graduates 
were employed full-time. 

^ . » • 

• 30% of those employed wefif^~^ployed in clerical 
and sales occupations, 21% in service occupations. 

* • 

The authors conclude that all six of their major voca- 
tional education a-eas are represented in the employment 
profile and that c though 26% of the graduates indicated 
thoy were unemployed, only 7% were looking for work. Spe- 
cific recommendations for the Laconia Area Vocational Center 
are made. 



This study was sponsored by the U. S. Department ot Health, 
Education, and Welfare. The senior author is an Assistant 
Professor of Education at Plymouth State College? the junior 
author is the Vocational Education Director of the Laconia 
Area Vocational Education Center #8. 



EVALUATIVE COMMENTS: 

Prom the stated purpose of the study, the reader would 
assume that the population consisted of graduates of voca- 
tional programs; however, this fact is not explicitly stated. 
Further, no differentiation in the various types of vocational 
courses taken by students was made. As a consequence, it is 
difficult to account for the reported employment rates. 

. Relying on sources other than graduates themselves to 
determine employment status posed ,a serious methodological 
problem. Further, there is ho indication of how much data 
were collected from secondary sources. Without some sort of 
verification, the data cannot be considered reliable. 

Since data were collected from graduates of several years, 
it is unfortunatie that some form of trend analysis was not 
done which would be particularly useful if it could be re-* 
latod to any curricular changes made in those years. 



* * * 



The Impact of Secondary School Occupational Education in 
Massachusetts . William G. Conroy, Jr., and Daniel E. 
Diamond. Lowell: University of Lowell, College of Man- 
agement Science, Spring 1976, 43 pp. (ED 122 095) 



DESCRIPTIVE ANNOTATION: 

According to published goal statements, Mass^-^^.vjhu setts is 
committed to providing youth with the skills ana experiences 
necessary for entry and success in the labor market. Because 
of this commitment and because of the wide variety of occu- 
pational education programs offered in Massachusetts, this 
study was conducted to examine the impact of high school pro- 
grams on the life style of students in college preparatory. 



general/academic, and occupational programs. It was 
intended to provide a knowledge base for improving secondary 
education policy. ' 

-The , study focused on school/ labor-market relations but 
also address..* citizenship and leisure-time activities. The 
conclusions drawn and recommendations made in this report 
are not based on' an exhaustive analysis of the data collected. 
However, the data are available from the Massachusetts Divi- 
sion of Occupational Education for anyone who may wish to 
explore the data base further. 

The research design was ex post facto: A sample of stu- 
dents who graduated (or "should have') from llassachusetts high 
schools in 1969 and 1973 were asked to respond to a question- 
naire mailed in the summer and fall of 1975. A multistage 
sampling plan was used: first schools, then students. A 
figure depicting the variables considered in the total' design 
is provided. Questionnaires were mailed to 7,894 students* 
2642 responded, 34 52 did not; and there were 1800 nondeliver- 
ables. If nondeliverables are assumed to be random, the 
response rate was 43%. Although the tindings are not cor- 
rected for bias, the data are weighted according to school 
s.ampling characteristics. 

The survey instrument is not discussed or appended; how- 
ever, categories of the questions asked are included in the 
figure depicting the study design. There is some discussion 
ot the operationalization measurement of the variables. Vot- 
ing behavxor is used as a "proxy" for citizenship. Salary 
data are presented in terms of purchasing power of 1975 dollars. 

Fisher t-tests, chi-squares, and correlation coefficients 
arc used to analyze the data and their uses are described. 
The data collected about major noneducational influences on 
liu used as controls so the independent effects of 

school programs could be estimated. Tables breaking the data 
down according to such variables as school program, grade 
point average, and postsecondpry education status are pre- 
sented. Among the eleven summary findings presented and dis- 
cussed are these: 

• There were no differences in average income 
between females who had occupational training 
and those who had none. 



65 



ERIC / 2 



• For students who did not go on to postsecondary 
education, the average annual salary of 1973 
occupational students was higher than that of 
nonoccupational students (a finding extremely 
pronounced for male Trade and Industrial stu- 
dents) . • 

• Two- thirds of occupational students did not 
work in fields for which they had trained. 

Among the conclusions discussed are these: 

• The superior labor mark^^t behavior of occu- 
pational, students seems to resul-t from "the 
school program as an independent entity 
because the effect exists, even after account- 
ing for variables such as socioeconomic status 
and scholastic aptitude. 

• Secondary schools play a minor role in help- 
ing youth find employment.. 

Six recommendations for Massachusetts policy makers are pre- 
sente<4. 

This study was initiated by the Division of Occupational 
Education, Massachusetts Department of Education. It was 
conducted by professors in the College of Management Science, 
University of Lowell. 

EVALUATIVE COWIENTS: 

This study is notable for the completeness of the infor- 
mation given, although it is unfortunate that no information 
regarding the survey instrument is provided. The design is 
clearly laid out; data were collected on inter*' -»ning vari- 
ables; and response bias was considered, although not analy- 
zed. Severdl aspects of the questionnaire can be inferred 
from the discussions of summary findings but there is no 
indication of its reliability or validity. 

Another notable aspect of this report is that of offer- 
ing readers an opportunity to further explore the data base. 
Not only does this allow analysis of data collected but not 
analyzed, but also it allows analyzed data to be reanalyzed, 
perhaps by use of different methods. 



* * * 
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"Intensity of Occupational Training: Its Effects on Subse- 
quent Labor-Market Experiences." Robert E. Allen and Thomas 
G. Gutteridge. Community/ Junior College Research Quarterly 
Vol. 2, pp. 367-380, 1978. ' ' 



DESCRIPTIVE ANNOTATION: 

This study was designed to investigate the relative ef- 
fectiveness of occupational programs of public and proprie- 
tary schools at the postsecondary level. Although both kinds 
of institutions are designed to develop job skills, they tend 
to differ in the proportion of job skills training and liberal 
arts education offered. Two other studies have investigated 
this issuers but neither controlled for two important vari- 
ables likely to affect postgraduate labor-market activity: 
geographic location and program quality. This study seeks 
to provide information which will enable the student to choose 
the type of program suitable to his/her specific abilities, 
needs, and objectives. The, data collected to provide this 
information concerned student career aspirations and work 
educational experiences before, during, and after two-year 
college attendance. 



The population comprised alumni who .had graduated from 
four upstate New. York two-year postsecondary institutions — 
two community colleges and two proprietary business insti- 
tutes. Because both proprietary schools had been in exist- 
ence for over 100 years and because proprietary schools are 
subject to extensive regulation in New York, the proprietary 
and public schools were assumed to be relatively homogeneous 
with respect to quality. 

The sample consisted of a random sample of 50 graduates 
from each school having a graduating class between 1962 
and 1970, thereby excluding the years 1965, 1967, and 1969. 

Questionnaires were mailed to 1773 male alumni and usable 
responses were obtained from 683 graduates (38.5%), but the 
number was reduced to make the sample more homogeneous by 
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S. M. Jung, V. N. Campbell, and J. M. Wolman, "A 
Comparative Study of Proprietary and Nonproprietary Voca- 
tional Training Graduates." Jo urnal of Vocational B ehavior 

18 (1976) 202-225. Wellford W. Wilms, Public and 

Proprieta ry Vocational Training (Berkel ey, Caiit. ; Center 
for Research and Development in Higher Education, 1975), 



excluding nonbusiness majors and graduates who had received 
baccalaureate degrees. The study sample, then, consisted 
of 173 proprietary graduates and 105 community college 
graduates. A five-minute telephone interview was conducted 
with a sample of the nonrespondents to assess nonresponse 
bias. Chi-square analyses indicated that respondents and 
*nonrespondents were not significantly different (.05 level) 
with respect to several variables. 

Ah eight-page ^questionnaire was used for this study. 
Its development is''^ not discussed, nor is it appended. One 
question asked respondents to indicate for each full-time 
job since graduation (1) the name and address of the employ- 
err (2) the scope of the business operation; and (3) the 
job title, functional area and annual compensation. Another 
question concerned what extent first and present jobs were 
related to training. Job satisfaction was measured through 
the Job Description Index developed by Smith, Kendall, and 
Hulin.29 

The labor-market experiences of the public and proprie- 
tary samples were examined at two time points; at initial 
entry into the labor market and at the time of the survey. 
Chi-squares, t-tests, and multiple regression were used to 
analyze different aspects of the data. 

Among the results discussed are these: 

• The type of institution attended did not 
greatly affect the measured n6neconomic 
aspects of the later labor-market experi- 
ences . 

• There was a sizable, but not statistically 
significant (at the .05 level) difference 
between present salaries of alumni of the 
two types of institutions, with those of 
community college graduates being higher. 

• Over 50% of the graduates were employed, 
both initially and at the time of the 
survey, in positions slightly or wholly 
unrelated to their training. 



^^r. K. Smith, L. M. Kendall, and C. C. Hulin, The 
Measurement of Satisfaction in Work and Retirement . (Chicago 
Rand McNallyT 1969) . 
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In conclusion, the authors discuss some of the limita- 
tions of this exploratory study. They acknowledge that it 
stressed investment rather than consumption values of educa- 
tion,, that the time frame was short compared to the life- 
long ramifications of education, and that the Ptudy was 
limited to four schools in upstate New York. Although they 
suggest the need for further research, they conclude that 
students can, apparently, "select a school with a pedagogy 
consistent with their educational objectives without jeopar- 
dizing subsequent career opportunities." 

The researchers are from the Department of Business Admin- 
istration, University of Wyoming, and the Department of Organ- 
ization and Human Resources, State University of New York at 
Buffalo. The research was conducted pursuant to a grant 
(No. 91-36-73-13) from the Employment and Training Adminis- 
tration, U. S. Department of Labor. 



EVALUATIVE COMMENTS: 

Although this study has limitations, including those 
acknowledged by the authors, it is notable for at least two 
reasons: (1) It builds on previous research, thus adding 
to an integrated body of knowledge and (2) An- effort was" 
made to assess response bias. The authors' suggestion for 
further research needs to be followed to lend support to thei 
conclusion that public and proprietary schools do not have 
differential effects on the career opportunities of students. 
This conclusion is unwarranted on the basis of a single 
study' because it is essentially an acceptance of the null 
hypothesis. •'0 



* * * 



See one of the research texts listed in the technical 
notes section of this bibliography for a discussion of this 
point. Very briefly, if research discovers a difference be- 
tween two groups. It is clear that a difference exists, al- 
though the cause may be questionable. If no difference is 
discovered, however, it may be because the observational 
techniques and statistical analyses were not powerful enough 
or were otherwise unsuitable for detecting a difference 



Job Selection Patterns^ Linkage Bfetween Vocational Educa - 
tion Programs and the Labor Market ^ George H. Copa and 
Bruce A. Kleveii. Minneapolis t MTnnesota Research 
Coordinating Unit for Vocational Education, Department of 
Vocational and Technical Education, University Qt Minne- 
sota, February 1977, 161 pp. (ED 138 809) 



DESCRIPTIVE ANNOTATION: 

The purpose of this study was to develop and test a con- 
cept for defining the linkage between vocational education 
programs and the labor market. This concept, termed "job 
sexection pattern," is defined as "a description of the jobs 
selected by program leavers in terms of frequency, occupation, 
industry, geographic location, and time." The authors state 
that meaningful planning of vocational education prograirts 
(i.e., planning that will insure gainful employment opportun- 
ities for program leavers) can be done on the basis of 
information concerning labor-market demand only if job 
selection patterns are identified and if the patterns are 
stable from year to year (or instability is explained) . 
Stability of job selection patterns refers to the degree to 
which graduates of a particular program select jobs in simi- 
lar occupations and industries in the same proportions from 
year to year. In order to test the job selection pattern 
concept on graduates of Minnesota area vocational technical 
institutes, the following objectives were accomplished: 

• Describing the pattern of jobs selected by 
graduates 

• Assessing the stability of the job selection 
pattern of graduates from year to year 

• Describing the job selection of graduates 
by geographic location 

• Assessing the stability between years of the patterns 
of geographic location of jobs selected by graduates 

• Assessing the longitudinal stability of the 
job selection pattern by graduates 

The information needed to complete this study had been 
collected through the Minnesota Vocational Follow-up System 
(a copy of a portion of the instrument is appended) . How- 
ever, all the information for every program had not .been 
coded or stored in the computer system. Therefore, a pur- 
posive sample of program areas with stored information was 
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selected. The criteria used for selecting the programs are 
discussed. The programs se3ected were as follows: agri- 
technology; automotive; clerical training; food preparation 
(chefs and cooks); electronics; machine shop; practical nurs- 
ing; and sales and business management. Samples of students 
graduating in 1970, 1971, and 1972 were chosen to restrict 
the sample size to 100-150 graduates of each program who 
were employed each year. Thus, for some programs, a random 
sample was taken, whereas for smaller programs, all workina 
graduates were included. ^ 

Graduates of 28 of Minnesota's 34 area vbcational tech- 
nical institutes were included in the sample. Demographic 
characteristics of the institutes and the students included 
in the sample are discussed, and a table is provided indicatinq 
numbers and percentages of students returning the 
follow-up questionnaire, and of those currently employed. 
The average return rate of the Minnesota Follow-up System 
over all postsecondary area vocational-technical institute 
graduates was approximately 80%. 

Most of the data analysis was accomplished through com- 
pilation of frequency distributions and determination of 
percentages; chi-squares were used to assess the statisti- 
cal significance of change in patterns over time. 

Results of the study, relative to each of the objectives 
listed in the first paragraph of this annotation, are dis- 
cussed by occupational program area. For each program area, 
a matrix is provided indicating percentages of graduates em- 
ployed in various occupations (e.g., professional, sales, 
blue-collar) in various industries (e.g., agriculture, manu- 
facturing, finance, entertainment). One of the findings 
discussed is the percentage of graduates over the three-year 
period remaining in the same major occupational and industry 
group from the first job to job-after-one-year (from 56 ^o 
59% for most program areas) . "i" 

Among the seven conclusions and recommendations listed 
and discussed are these: 

• There is a characteristic pattern of jobs 
selected by graduates of vocational education 
programs unique to each program and with 
overlap between program areas. 

• The job selection pattern changes during the 
first year after graduates leave vocational 
education programs, and the degree of change 
varies between programs. 



• Follow-up data on vocational graduates should 
include the information necessary to construct 

job selection patterns in order that the ^ • 

utility of the concept may be further assessed. 

• Studies to explain the unstable aspects of 

job selection patterns should be initiated, — 
in order to indVease the utility of projected 
information concerning labor-market demand 
and to identify means of effecting needed 
changes in selection patterns. 

EVALUATIVE COMMENTS: 

The stated purposes of many vocational education evalua- 
tion studies indicate that the studies are done to provide 
information for planning purposes. The authors of this re- 
port state that information about job selection patterns 
can increase the validity of planning that is based on in- 
formation concerning labor-market demand, but that data re- 
quired to compile such information are rarely available. 
The authors acknowledge that further exploration of the Dob 
selection pattern concept is necessary, and this document 
makes a convincing casf for the desirability of collecting 
the data necessary for such further testing. 

AAA 



"Little Woolly Horses and Saber-Toothed Tigers: The Mis- 
guided Vocational Education Curriculum." Everett Egainton.^ 
Louisville: University of Louisville, n.d. , 42 pp.**-*- 

DESCRIPTIVE ANNpTATION: 

The intent of this study is "to examine the relation- 
ship between enrollment in vocational education programs 
and attitudes toward self and toward education." Specifi- 
cally, it addresses the question "Does participation in 
vocational education programs result in positive attitudinal 
changes among students?" The author consider? this questio'i 
of general interest because there is "a lack of evidence 



^^No further citation is given on this unpublished paper; 
however, an article based on it appeared in Phi Delta Kappan 
(April 1978, pp. 533-534) under the title "Is vocationax 
Education Meeting Its Objectives?" 
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to support educators' claims about the efficacy of vocation- 
al education programs..." and specifically' because "aliena- 
tion among students in vocational education programs. .. is 
at an all-time high even though positive attitudes toward 
self, toward education and ultimately toward work are un- 
questioaably important goal% of virtually allvvocatiional 
education programs." The specific hypotheses' tested are 
tvrofoldt (1) Vocational education students have'' a more 
negative self-image than nonvocational or academic students. 
(2) Vocational education students have a more negative 
attitude toward learning than nonvocational or academic ^ 
students. 

♦ 

The students constituting the sample were selected from 
public high schools in Kentucky through a proportional 
stratified random sampling technique. Schools were randomly 
selected from the state's fifteen vocational education 
r.egions, and a questionnaire was administered to 'approxi- 
mately 50 per cent of the juniors and seniors from each 
selected school. Valid responses were obtained from 8145 
students. Two possible biasing factors were mentioned: 
(1) Principals and guidance counselors, rather than re- 
searchers, drew the samples and administered the question- 
naires; thus, improper .sampling may have occurred and/or 
inadequate instructions regarding the questionnaire may 
have been given to the students. (2) Some of the schools 
randomly selected were unwilling to participate, thus 
the study sample may not have been representative of the 
population. 

As part of a project directed by Dr. Keith Bayne 
(a professor pf Vocational Education at the University of 
Louisville) , a 38-item multiple choice questionnaire was 
administered to the above-described sample between September 
1 and December 15, 1976. The study here annotated consists 
of Egginton's ^nalysis of some of those questionnaire items. 
The purpose for which Bayne developed the questionnaire is 
not discussed; however, the instrument is appended. 

The independent variable — enrollment in a vocational 
program — was defined by answers to the question: "Are you 



Tho total public high school enrollment in Kentucky 
m 1976 was 216,009 (U.S. Bureau of the Census. Statistical 
Abstract of the United States : 1977 (98th editiorvJ 
Washington, D.C., 1977)]. Thus, over 3.7% of the juniors 
and seniors were sampled. 
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presently enrolled in a vocational program (examples: busi- 
ness ana office, distributive education, health occupations, 
agricultural education, etc.)?" All students who answered 
yes were classifified as vocational students and the others 
as nonvocational. General and college-preparatory students 
were not separately analyzed. 

The dependent variables — self-image and attitudes toward 
education— also were measured by responses to specific items 
on th6 student-survey questionnaire. Two questions, each re- 
quiring a yes or no answer, were used to assess two aspects 
of self-image — def ensiveness and inadequacy: "Do you usually 
feel like the world is against you?" and "Do you usually feel 
that most people do things better than you do?" The question 
used to assess attitude toward learning was not identified, 
but presumably it was "How do you feel about learning?" The 
choices were (1) like learning most of the time, (2) like 
learning some of the time, (3) like learning only things of 
interest, and (4) dislike learning most of the time. 

Four control variables are used in analyses of the hypo- 
theses: sex, age, ethnic origin, and family income. Although^^ 
the questionnaire elicited more precise information, the lat- 
ter three variables were artificially dichotomized: 16 and 
under — 1^ and over; white — nonwhite; $9,000 or less per annum — 
more than $9,000 per annum. « 

The data are all reported as cross-tabulations, including 
number of respondents as well as percentages; missing values 
and missing cases are reported. The phi coefficient of 

correlation was used in all analyses and the attained signif- 
icance level is reported. 

Among the results reported and discussed are the follow- 
ing : 

• "Participation in the vocational education curricula 
is apparently associated with feelings of defensive- 
ness, although the relationship is slight." (j^ « .02) 

• "The association between vocational education curric- 
ula and feelings of def ensiveness holds — and indeed 
is strong — for students 16 and less but virtually 
disappears for students age 17 or more." (f^ = .^5 
and .007 respectively) 

• '"Participation in vocational education curricula is 
associated with feelings of inadequacy, although, as 
was the case with defensiveness, the relationship is 
slight." (jzJ = .02) 
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• "When controlling the age^ however, the relationship 
(between enrollment in vocational programs and feel- 
ings of inadequacy) is not sustained for either the 
younger students or the older Students." (ji » .02 
in both cases) 

• "There is a significant and inverse relationship be- 
tween enrollment in vocational curricula and [posi- 
tive] vattitude toward learning." (0 « ,05) 

The author concludes "... vocational education programs 
in general are ineffective in changing ( negitive attitudes and dif- 
ferences between vocational education students and their aca- 
demic counterparts] ." He states, "Despite the massive invest- 
ment of funds in vocational education programs and despite 
the recognition that vocational education curricula should 
address the problem of negative values and attitudes, the 
vocational education student still holds himself in low es- 
teem and tends to treat learning with disdain." 

Egginton then makes several recommendations not linlced 
to the data analyzed in this study. For example, he states, 
"In order to begin to combat the problem of negative atti- 
tudes, low self-image, and disdain for learning among voca- 
tional education students, the larger society must provide 
the prestige, status, ^nd remuneration which will make voca- 
tional education seem worthwhile to the student." Further, 
ho supports views attributed to Terrel Bell, "Different 
and separately adminic wored academic and vocational programs 
serve no useful ends"; to M. E. Borus, "On-the-job training, 
apprenticeships, and other non- formal methods of vocational 
training , , , are more efficient and cost-effective than the 
vocational training provi^^ed by schools"; and to P. J. Kelly, 
"The most important job skills which a high school graduate 
can possess are those provided by general education." 

The author of this report is a professor at the Univer- 
sity of Louisville. It is stated that Kentucky was chosen 
for this study because its "posture toward public vocational 
education programs appears to be representative of the 
willingness of many of the states to establish vocational 
education as a top priority in public education," and 
appreciation is given to the Kentucky Bureau of Vocational 
Education for permission to analyze the data, but no 
furthef indication of sponsorship for this study is given. 



EVALUATIVE COMMENTS: 

Although this study is notable iv that, unlike the 
other studies list^ed in this bibliography, It addresses an 
immediate impact of vocational education , it is fraught with 
so many methodological problems as to make its concluaions 
unwarranted. This is not to say that the conclusions are 
necessarily inaccurate, but rather that they are neither sup- 
porter: nor contradicted by the study. 

The instrument used in the study is appended, but its 
development is not discussed. No information is given con- 
cerning its reliability and validity. Even among the 
valid responses collected, each question analyzed in the 
study was answered by only one to three percent of the 
btudents. Further, no rationale is given for jua.ing de- 
fensiveness, ihadequacy, and attitude toward learning on 
the basis of one questionnaire item each. 

Eggintoh hopes to answer the question "... Does 
participation in vocational education programs result in 
positive attitudinal changes among studc^nts?" His * -ign, 
however, does not allow for analysis of change. No argu- 
ment is made to support the assumed comparability of 
vocational high school students and all other high school 
students in terms of factors that may affect self-image, 
there is no attempt m^de to analyze statistically the 
collected data that best suits the question of change — 
namely, the attitudes of sophomore vocational students as 
compared to thosv of senior vocational students. 

Another important problem is that of statistical sig- 
nificance as opposed to meaningful significance. Although 
several of the relationships analyzed were found to be 
statistically significant, the largest coefficient of de- 
termination obtained was .0098.^^ ■ This coefficient indicates 



33Although it is stated t;-»at juniors and seniors were 
sampled, data tables indicate that .sophomores were as well. 
Even if there were too few sophomores for meaningful com- 
parisons, an exploratory comparison could have been made, 
accompanied by a suggestion for further data collection. 

34see one of the research texts listed in the technical 
notes section of this bibliography for a discussion of this 
noinfc Briefly, the square of a correlation coefficxent is 
i;Se"coef?Icient of determination. When multipled by 100, 
the result indi9ates how much of the variance in one corre- 
late is explained (not necessarily caused) by the other. 
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that less than 1% of the variance in any of the variables 
Studied can be accounted for by any other variable. The 
meaningfulness of this small amount of variance is questionable. 

Egginton draws conclusions that do not follow :from his 
data. For exanple, he states, "Table 3 demonstrates that 
■the association between vocational curricula and feelings of 
defensiveness holds— and indeed is strong-for students age 
sixteen and les? but viituallv disap pears for students aal 
seventeen and more" (emphasis ade ledy. First, the phi coef fi- 
cient of .05149 that he reports cannot be said to indicate a 
strong association (it explains only one quarter percent of 
the variance) . Second, he later concludes that vocational 
nrHf?on«^f,r ^ negative self-image (defined in terms 

of defensiveness and inadequacy) than nonvocational 
students, and "these group differences are susta ined for the 

older . . . students" (emphasis added) . He also makes 

recommendations 'without clearly stating that they are not 
direcitly based on the research reported in his paper. For 
example, he states that on-the-job training is more efficient 
and cost-effective than vocational education provdded by the 
schools, although he reported no data regarding relative 
effectiveness of various vocational training methods.^ 



* * * 



19 75 Career Student Follow-up: Initial Placement . Michael 
Quanty!. Overland Park, Kansas: Office of Institutional 
Research, Johnson County Community College, April 1976. 
25 pp. (ED 126 999) 



DESCRIPTIVE ANNOTATION: 

Johnson County Community College (JCCC) is required to 
report employment data on its graduates and on students who 
leave with marketable skills to the K^r^as State Department 
of Education and the Kansas Manpower utilization System for 
Training. These required data are uriel to justify expendi- 
ture of state and federal vocationa». funds on career pro- 
grams. The college takes the oppcr ..unity provided by this 
mandated data collection to acquire additional data tor use 
in Its ongoing self -evaluation process, 

t,-iT^? graduates and 41 nongraduates with "marketable 
skills who left ten career programs (Basic Police Training 
Data Processing, Dental Hygiene, Drafting, Electrical- 
Electronics Technology, Fashion Merchandising, Law Enforce- 
ment, Marketing and Management, Nursing, and Secretarial 
Career 3) during the 1974-75 academic year constituted the 
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basic population for thifs dtudy. Responses were obtained 
from 94% of the population » and alsb from seven students 
from the program for the hearing impaired, two of whom were 
also enrolled in career programs. 

Most of the data were collected through telephone 
interviews, although the students with hearing impairments 
were contacted personally. Career program coordinators 
telephoned graduates and early leavers from their respective 
programs; when the graduate or leaver was not available, 
the coordinator attempted to obtain information from a 
close relative or work associate. The telephone interview 
guide is appended. Questions addressed employment status 
(whether employed in program area, in a related area, or 
otherwise employed; whether attending school in a training- 
related area or in another area; unemployed; deceased or 
disabled; in the armed forces); adequacy of training; and 
gross annual and monthly salary. Additional questions con* 
cerning the specific programs were asked by some of the 
coordinators (e.g., concerning interest in alumni group 
participation, or in additional coursework) . 

The data are reported as percentages, both ^or the total 
sample and by program; no statistical analyses were per- 
formed. In addition, some data collected in th& previous 
three follow-up studies are provided but no comparisons 
are made. 

Several results are discussed. Of the former students, 
70% were employed in a job for which they had been trained 
and 17% were in a training-related job. For two of the pro- 
grams, students had to be employed before admission; and, 
excluding these students, 63% and 21% respectively were so 
employed. Of those employed in training or training-related 
areas, 67% believed their training helped them obtain their 
jobs. Overall, 93% rated the JCCC program as either excel- 
lent or good, and at least 80% of the students in each pro- 
gram gave excellent or good ratings. The two students with 
hearing impairments who had been in career programs (draft- 
ing, electronics) were employed full-time, but neither in 
(the area of training. No interpretations are made or con- 
clusions drawn from the results reported. 

This report was done by a member of the, college's Office 
of Institutional Research. According to the title page, 
it was prepared for administrative review. 
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EVALUATIVE COMMENTS; 



Presumably, because the report was prepared for admin- 
istrative review, it was purposefully left to the adminis- 
trators to interpret results and draw conclusions. More 
information would facilitate this effort, for example, why 
some students do not find training-related employment and 
what particular aspects of their programs helped students 
obtain jobs — their diploma or the skills they had obtained. 
To allow such interpretations to be made, additional data 
nped to be collected. 



* * * 



1000 Empl oyers Look at Occupational Education . Martin Ham- 
burger and Harry E. Wolfson. New York: Board of Educa- 
tion of the City of New York, July 1969, 216 pp. 



DESCRIPTIVE ANNOTATION; 

This was the first in a series of studies planned to 
provide a sound base for curriculum redevelopment in the 
occupational education programs of the New York City Board 
of Education. The objectives of the overall plan to make 
fundamental changes in occupational education were as 
follows: (I) to improve the curriculum; (2) to extend occu- 
pational education to more youth; and (3) to introduce a 
multilevel approach with a variety of time and sequence 
organizations. 

This study was designed to survey employers in an 
effort to determine how to prepare a greater number and 
variety young people for better jobs. Effective ques- 
tions to be asked were discussed in workshops — one composed 
of leaders from industry, commerce, labor, and education; 
the other composed of administrators, supervisors, and 
teachers in the New York Citv Schools. 

This project was conducted from February 1 to June 30, 
1968, following a long planning period. From all the occu- 
pational areas in New York City, a sample of curricular 
areas was selected based on diversity, importance, projected 
growth, and adaptability to a variety of school plants ana 
organizations. The five at eas selected — business, health. 



automotive* metal-working, and electrical/electronics— 
include predominantly male, predominantly female, and mixed 
occupations. They include also a range of skill levels ... 
a balance between white and blue collar, between industrial 
and service, and between stable and rapidly changing fields. 
Firms of various sizes were selected from each category. 
Interviews were held with 1056 employers, with usable data 
obtained from 994. Because the final sample was not losen 
throusih a rigorous sampling plan, but rather was base^l on 
the criteria stated above * nd on availability and coopera- 
tion, the authors note that ^hat was done was a case study 
of selected employers rather than a study of a random sample. 
Interviews were held with presidents, managers, personnel 
directors, and production supervisors. 

Project staff prepared an open-ended questionnaire and 
interview guide but, after using it, decided that a more 
objective approach would be more fruitful. The instrument 
was then redesigned. The redesigned questionnaire, which 
is provided in an appendix, basically follows a checklist 
format, although open-ended comments related to each 
questior. r.re sought as well. Included are questions about 
job needs, job titles and duties, pay levels, desired 
educational levels, etc. 

The instruments were used during team visits to each 
employment site. Five teams were selected, each including 
one teacher and one guidance counselor. Team members were 
assigned full-time to the project and were released from 
their teaching and counseling responsibilities fot its 
duration. An intensive three-week training period was 
held, followed by weekly staff meetings during the begin- 
ning of the data-collection phase. 

The data analyzed for this study consisted of responses 
to the survey questionnaires and impressionistic statements 
written by the interviewers. The interviewers also wrote 
recommendations for curriculum development. Quantitative 
data were synthesized into question-by-question summary tables 
and qualitative data w4re content-analyzed so they could be 
condensed. Excerpts from comments and recommendations are 
presented, both in the text and in appendixes. 

Findings are presented primarily by occupational group- 
ings in both a detailed chapter and a summary chapter. Among 
the findings discussed are these: 

• Contacts between schools and employers tend 
to be "hit-or-miss." 
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• Employers indicate that the kind of reading skills 
tauqht, such as reading work orders and technical . 
manuals, is as important as the amount taught. 

• Although a substantial minority of employers 
consider on-the-job training best, for the 
most part employers feel that a school-job 
partnership is optimal for occupational 
training. 

These and other findings are discassed_in detail? many tables 
are presented; ar 1 limitations of the data are discussed as 
well ds their imi ications. 

This study was funded under the Vocational Education 
Act of 1963 and was approved by the Board of Education of 
the City of New York and the New York State Department of 
Education. One principal researcher was a Professor of Edu- 
cation at New York University and vhe other was Assistant 
Superintendent, Office of High Schools, New York City Board 
of Education. ^ 

EVALUATIVE COMflENTS: 

This study was conducted to collect employer data for 
oo^f .''"^''i?"^''"* development. It can thus be viewed in the 
context of discovery rather than as an attempt to verifv anv 
hypotheses about the outcomes of vocational ^ducatio^Thr 
report is exemplary in being very thorough: The pirSise 

an^^y^'L^I'dlta'and'^i"''""^^^'^ .^^^^^-^^ 

Ind LstrumonL'aon^rl'd^tS?!^ procedures 



* * * 



Outcomes of Vocational Education in Virginia lD7b: A 
S ummary of the Follow-up of 1976-1977^ S econdar v Voca- 
tio nal Education Cuinplotions in Vi-'gini a. Donald E. 
Elson. Richmond: Virginia u€»partment of Education, 
Vocational and Adult Education, 1978, 18 pp. 
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DESCRIPTIVE ANNOTATION: 



The information reported in this document was collected 
as part of the annual follow-up begun when the Vocational 
Education Reporting System (VERS) was implemented in 
Virginia during the 1972-73 school year. Data were 
collected from former students concerning employment status, 
on-the-job use of knowledge and skills learned in vocational 
programs, wage rates, and perceptions and attitudes 
.toward their vocational programs. No specific research 
questions or hypotheses are listed. 

This study was conducted in January 1978, approximately 
seven months after the close of the school year. The , 
population was Virginia public high school students who 
had completed a vocational education program during or 
after the 1976-77 school year or who had left school with 
marketable skills before completing their programs (how 
their marketable skills were determined is not stated) . All 
business education students and a stratified sample of 
students from other vocational programs were studied. 
Tables indicating total completions in each area, sample 
size, and number of usable responses are provided. The 
overall response rate was 55%, representing 10,961 usable 
responses from the 19,869 students surveyed. Response bias 
was analyzed using chi squares on various demographic char- 
acteristics. The data are summarized in a table. There are 
two statistically significant differences: (1) More females 
than males responded. (2) Proportionately few^r students who 
had left school prior to scheduled graduation returned ques- 
tionnaires than students who were completers. Development 
of the instrument is not discussed, nor is it appended, but 
the several tables and the discussions clearly indicate the 
nature of the questions asked and the form of the response. 
Reliability and validity information is not provided. 



Percentage data for all questions asked are presented in 
"pie charts" and bar graphs for each vocational program 
area. No statistical analyses were done. Comparisons are 
made to the 1975-76 school ye^r. 

Among the findings discussed are the following: 

• 39.15% of the respondents were employed full-time 
in the field for which they were trained, or in a 
related field. 
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• 82.76% of the respondents were emnloyed full- or 
part-time, wore continuing their education, or 
were in the military service. 

• 25% of the respondents were continuing their edu- 
cation. 

It is concluded that vocational education programs are 
effective in helping former students to obtain employment. 

The Vocational Education Reporting System was developed 
cooperatively by the Virginia Division of Vocational and 
Adult Education and the Division of Vocational and Technical 
Education, Virginia Polytechnic Institute and State Uni- 
versity. The author is a faculty member. Staff of the 
Virginia Department of Education contributed to the data 
analysis and development of the report, 

EVALUATIVE COMMENTS: 

Although a purpose for this study was not stated, it 
can be inferred from the conclusion that the purpose was to 
assess the effectiveness of Virginia's public secondary 
vocational programs in helping students obtain employment. 
The follow-up system reported here may be capable of 
yielding accurate descriptive information about what has 
happened to former students, but it does not collect 
sufficient data from which to infer causes. In order to 
ascertain whether students obtained employment because of 
the program or some other factors or whether failure to 
obtain employment can be attributed to the vocational 
system, comparisons must be made and other data collected 
such as unemployment rates in general in the community, 
unemployment rates of nonvocational students, and availa- 
bility of jobs. 

Regardless of the quality of the reporting system, the 
response rate for this particular study was too small to 
ensure that the study sample was representative of the 
population. Although response bias was analyzed through 
use of demographic characteristics, there is no assurance 
that other factors, such as employment status, did not in- 
fluence tendency to respond. 

The ability of students to accurately assess the use- 
fulness of their training is questionable, as is their 
tendency to give an honest and thoughtful response to a 
questionnaire item. Other data could be collected in an 
attempt to verify the conclusion that training is helpful. 
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For example » employers' opinions could be sought or employed 
graduates of nonvocational programs could be asked whether 
they found their lack of training to be a disadvantage. 



* * * 



"Outcomes of Vocational-Technical-Transfer Programs at 

Community Colleges, Technical Schools, and Similar Types 
of Institutions." Richard J. Noeth and Gary R. Hanson. 
Paper presented at the Association for Institutional 
Research Annual Forum, Los Angeles, California, May 
1976, 16 pp. (ED 1^2 892) 



DESCRIPTIVE ANNOTATION: 

This study, conducted in 1975, was designed to assess 
the outcomes of postsecondary vocational-technical- 
transfer programs, specific questions were addressed: 

• What is the present educational<»vocational status 
of individuals who began vocational-technical- 
transfer programs at community colleges, technical 
schools, and similar types of institutions in the 
fall of 1970? 

• How do students' final programs relate to their 
occupational experiences and to their future occu- 
pational plans? 

• Do individuals working in program-related occupa- 
tions feel that their postsecondary training 
affected the level of their present employment? 

• Are individuals working in program-related occupa- 
tions satisfied with their present jobs and would 
they go through the seune program again? 

In the fall of 1970, 22,342 students from 110 community 
colleges, technical schools, and similar institutions across 
the nation completed the American College Testing Program's 
Career Planning Program as part of the national norming 
(of ability, interests, job choice, job values, etc.). 

In 1975 a follow-up. sample of 4,350 former students 
was selected from the above group. For this sample, male 
and female students were selected from Business and Market- 
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inq, Accounting, Science, Social Science, and Arts and 
Humanities programs; only males were selected from Elec- 
trical Engineering Technology and Auto Mechanics; and only 
females from Nursing programs. A final response rate of 
60% — equivalent to 95% of those for whom accurate addresses 
were available— was achieved with 2594 former students 
from 109 institutions responding. 

The survey questionnaire was pretested extensively 
before it was mailed to the student sample. Its development 
is not described nor is the instrument appended. 

The data from all former .students who responded to all 
the items pertinent to each ana],ysis were cross-tabulated — 
vocational programs (Business, Auto Mechanics, etc.) by 
.criterion measures (present occupation, job satisfaction, 
etc.). Several tables are provided. 

The following findings related to each of the four 
study questions are discussed: 

# Among the respondents, 1,489 were employed 
outside the home, 234 were continuing their 
education, 124 were homemakers, and the rest 
were unemployed for various reasons. 



% Most of those currently employed held jobs 
for which they were trained (percentages 
are provided by program area) . 

• Over 75% of the currently employed individuals 
agree that they could not have obtained their 
present jobs without their postsecondary education. 

Among the major conclusions discussed are these: 

• About 75% of currently employed students are em- . 
ployed in their area of training. 

• Students tend to gravitate toward jobs related to 
the educational program they have completed, 
although their first job may not be directly related. 

• Those employed in training-related jobs are highly 
satisfied with their present occupations and the 
majority would enter their training programs again, 
if faced with a choice. 
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This study was conducted by a research psychologist and 
the Assistant Director of the Developmental Research De- 
partment, Research and Development Division, The American 
College Testing Program* 

EVALUATIVE COMMENTS: 

This study cannot- be replicated from the information 
given, but since it is a repo/t Of a paper presented at a 
conference, that is not to b# expected. Cross tabulations 
of data are provided but bedause statistical analyses are 
not, it is not clear how li|tely it is that differences are 
due to chance. 

The stated purpose of this study is to assess outcomes 
of post-secondary vccational-technical-transfer programs. 
It is unfortunate that the reason behind the purpose is 
not stated. It is difficult to further evaluate the study 
without knowing how it was meant to be used. It does give 
art indication of the average ( nationwide) success of the 
programs studied, but sufficient information for further 
conclusions is not provided. 

* * * 



A Program Review of Secondary Vocational Education in Ohio: 
Job Placement and State Funding . Ohio Legislative 
Services Commission, Staff Report No. 126, April 1978, 
73 pp. 



DESCRIPTIVE ANNOTATION: 

This study was done to provide feedback on the results 
of earlier vocational education policy and funding decisions 
and to provide useful information and guidelines for future 
legislative decisions. The study focuses on the job place- 
ment of vocational graduates (at the secondary level) and 
on related issues such as graduate job satisfaction and 
graduate performance as evaluated by employers. The issue 
of how state funds can be better distributed to meet pro- 
gram costs is also addressed but will not be discussed in 
this annotation. 

Ten vocational education planning districts (VEPDs) were 
randomly selected from the 65 that had graduated their first 
senior class by 1976. Four-day field visits were made to 
schools in these districts and one-day visits were made to 
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Akron City Schools and to a Joint /ocational School in the 
Cblumbus area {on the recommendation of the State Vocation- 
al Education Division that they had excellent placement 
programs) . One hundred sixty-six employers were inter- 
viewed by phone. Formal questionnaires were administered to 
vocational instructors and to placement coordinators, as 
well. The sample districts are described in an appendix. 

The questionnaires used i he staff field visits are 
not discussed in the report, a' ► ough they are appended. 

Findings -e presented in narrative form in tables of 
percentages and graphs of trends from fiscal year 1974 through 
fiscal year 1976. Comparisons on job placement are made 
among VEPDs and among vocational programs. Employers, where 
possible, compared vocational graduates to graduates of 
general programs. 

The results are discussed in great detail, including 
consideration of intervening factors such as availability 
of employment. The following findings are reported: 

• Vocational programs similar to those offered by 
technical colleges or proprietary schools do not 
have good placement records. 



• More than 90% of the graduates indicated that they 
would enroll ii. vocational education again. 

• Students and teachers expressed concern with tech- 
nical skills, whereas employers stressed employability 
sxills and adaptability. 

• 58% of the employers hire vocational graduates for 
jobs that nonvocational high school graduates 
cannot fill without further training. 

• 76% of employers said they prefer to hire vocational 
graduates rather than general high school graduates. 

• A few students in each sample district said they 
would have dropped out of high school had there not 
been a vocational program. 

Two conclusions reached are that the vocational education 
program in Ohio has been relatively successful in job place- 
ment and that employers, while generally satisfied with 
the skills of vocational graduates, would favor increased 
coirjnunication with school officials. The need for a closer 



87 



look at guidance and counseling and at the effects of 
vocational screening is suggested, and other areas for 
further research are indicated. 

In 1^77., the Ohio Legislative Services Commission 
selected a Subeommittee on Legislative Oversight; it sub-- 
sequently recotitmended education as one area for joint 
legislative-staff oversight projects. Staff and legislators 
received training and technical assistance in program review 
from Rutgers University's Eagleton Institute of Politics. 
This study was one of the projects undertaken. It was re- 
viewed and monitored by the Joint Education Committee for 
program Evaluation. 



EVALUATIVE COMMENTS; 

This study was well defined and thoroughly conducted. 
It is one of the few studies that compare perceptions of 
teachers, students, and employers. It is notable also 
for its discussion of economic conditions in the labor 
market and other variables that affect placement rates. 
Lacking are (1) a discussion of the development of survey in- 
struments and (2) the collection of placement rate data for 
nonvocational graduates. 

* * * 



A Study of Community College Students Who Are Graduates of 
Vocational Technical and College Preparatory Curriculums . 
Ronald J. Horvat-h. Schnecksville: Lehigh County Commun- 
ity College, Pti^nsylvania Department of Education, Bureau 
of Vocational, Technical and Continuing Education, July 1, 
1973, 96 pp. (ED 096 554) 



DESCRIPTIVE ANNOTATION: 

Although most community colleges maintain an open-door 
admissions policy, there are varying interpretations of the 
concept. Some colleges limit admissions to students who 
have completed high school college preparatory curricula, 
whereas others also admit students who have completed 
vocational curricula. The author states that decisions on 
interpretation of the open-door concept are seldom based on 
empirical data. This research focuses on the issue in an 
attempt to provide teachers, counselors, and administrators 
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with better insights into the needs and background of 
students (e.g.. Should students take vocational or college 
preparatory curricula if they intend to enter community 
college, and should they enroll in trannfer or occupational 
programs?). Specifically, the following three questions 
are addressed: 

• Are community college students who are graduates of 
vocational technical programs similar to or different 
from graduates of college preparatory programs in 
intellective and demographic characteristics? 

• What are the relative completion or noncoropletion 
records of college preparatory graduates as compared 
with those of vocational technical graduates? 

• What are the intellective and demographic character- 
istics of the vocational technical graduates who en- 

. roll in a related college curriculum, in an unrelated 
one, or in a transfer program? 

Six hypotheses relative to these questions are investigated. 

All male students in two Eastern Pennsylvania community 
colleges who could be identified as having completed one 
year or more in a high school vocational technical program 
(excluding commercial, general, industrial arts, and dis- 
tributive programs) constituted the vocational . technical 
(VT) sample; 229 were identified. The college preparatory 
(CP) sample consisted of a random sample of 200 male 
graduates prorated among the entry years of 1967, 1968, 
1969, and 1970, using the number of VT entrants as a base. 
This represented 10-15% of the full-time male population. 
Approximately 90 percent of the students in the samples were 
graduated from high schools within the sponsoring districts 
of the two colleges (included are four vocational technical 
secondary schools and no comprehensive secondary schools) . 

The primary sources of data were the students* college 
transcripts and personal files, which containet-^ hi5h school 
transcripts, class ranks, IQ scores, and American College 
Test (ACT) scores. 

Demographic differences and graduation and program 
associations were t*=sted with chi-squares. The primary data 
analyses of group differences were aticomplished through 
univariate analyses of variance (t-tests and P-te^ts) . A 
.05 level of significance was used in interpreting the 
results. 



o 

ERIC 



89 



^6 



Although CP graduates scored significantly higher than 
VT graduates on six of the seven educations i variables 
tested (the exception was high school rank) and achieved • 
significantly higher first-semester and final grade point 
averages, there was no significant difference in college 
J' graduation rates for the two groups. Therefore, the author 
concludes that this study adds a degree of credibility to 
the community college open-admissions policy. Other re- 
sults, pertaining to the different community college 
programs VT graduates enter, are discussed and areas for 
further research are suggested. 

The author is affiliated with Lehigh County Community 
College, one of the colleges in the study. Apparently, the 
sponsor of the study was the Pennsylvania Department of 
Education, Bureau of Vocational, Technical and Continuing 
Education. 

EVALUATIVE COMMENTS: 

An important strength of this study is that it had a 
specific purpose and hypotheses related to other research, 
as indicated in the literature review section. Thus, the 
study adds to a body of knowledge rather than being an 
isolated work. Further research is clearly needed, however. 
For instance, the relative impor'tance and consequences of 
community^^ college grades and community college graduation 
should be investigated. Following that determination, 
correlates of grades and graduation that are more specific 
than the general vocational technical as opposed to college 
preparatory high school curricula should be investigated. 

* * * 



A Study of the Costs, Benefits, and Effectiveness of Occu 
pational Educatio"n T Austin D. Swanson. Buffalo: 
Faculty of Educational Studies, State University of 
New York at Buffalo, March 1976, 96 pp. (ED 120 523) 



DESCRIPTIVE ANNOTATION: 

This study was conducted to provide the Board of 
Cooperative Educational Services (BOCES) for the district 
serving nine component school districts south of Buffalo, 
with "evidence as to vfhat, if anything, was being accom- 
plished [oy its vocational programs! that was beneficial to 
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the region." Fiye evaluative criteria had been previously 
established by the Board and its professional staff, and 
two were added for this study: 

• Upon completion of an occupational program, 65% of 
the students will indicate that they would take the 
oame program again if they were making the choice. 

• 75% of the graduates of any licensed occupational 
program will pass the appropriate examination. 

• Within six months after graduation 60% of the 
graduates available for and having sought employ- 
ment will be employed full-time. 

• Within six months after graduation, 35% of the 
graduates available for and having sought employ- 
ment in their specialization vrill be employed 
full-time in their area of specialization. 

• Within six months after graduation, another, 15% of 
the graduates available for and having sought employ 
ment in their specialization will be employed full- 
time in a position they consider related to their 
area of special ization . 

• The high school dropout rate will show a decline 
over 'the years. 

A high school education will prepare graduates to 
participate successfully in a wide variety of con- 
versation toDics and in leisure and civic activities 

This study was designed to address those criteria and to 
accomplish the following obj.actives as well: 



• To compare per pupil costs of vocational and non- 
vocational nigh. school programs in the Buffalo, New 
York, metropolitan area 

• To compare the relative success of vocational 
students and nonvocational students in terms of 
employment and earnings and also in terms of 

selected noneconomic considerations (on the basis 
of the established objectives listed above) 

• To calculate benefit/cost ratios for vocational 

proqrams 
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• To develop decision matrices for evaluating likely 
costs and effectiveness of alternate approaches for 
meeting the district objectives for occupational 
education 

The three phases of the study, conducted during the 
1974-7 5 academic year, were (1) a program cost analysis, 

(2) collection of effectiveness and control data, and 

(3) cbst/benef it-cost/effectiveness analyses. Costs were 
determined on a per pupil basis for the 1972-73 school 
year. Effectiveness data were sought from the 628 partici- 
pants of BOCES occupational programs who were members of 
the classes of 1969, 1971»and 1973 and from a random sample 
of 422 non-BOCES, noncollege bound^S students from the 
classes of 1967 (prior to establishment of the BOCES pro- 
gram) , 1969, 1971, and 1973. Usable responses were received 
from 61.5% of the BOCES alumni and 55.2% of the non-BOCES 
alumni. A table indicating response rates from each class 
is provided* School and background characteristics were 
analyzed for respondents and nonrespondents to determine 
whether ther« were any significant differences between the 
two groups. The only statistically significant difference 
found was between the Occupational Grade Average of BOCES 
respondents and that of nonrespondents; since the averaqes 
were 79 and 78., respectively, it is doubtful that the dif- 
ference has practical significance. 

The study design and methodology are thoroughly de- 
qcrihed as are the methods of analvris. For example, the 
difference between expenditures and costs is defined: 
"Expenditures refer to the dollar value of resources 
purchased during a given time period ... (whereas! costs 
refer to the amount of resources consumed during a time 
oeriod." As another example, it is stated that average 
teacher salary figures are used to preclude a bias against 
programs that happened to be staffed with more experienced 
teachers . 

Two instruments were used to collect effectiveness 
data, and both are appended. The School Record form was 
used to consolidate data taken from school files including. 



-'^Noncollege bound students were those for whom no 
high school credentials were sent to postsecondary institu 
tions durintT or shortly after their senior year. Approxi- 
mately 20% of these students did attend a postsecondary 
institution, but "long after leaving high school." 



for example, grade average^ class rank, IQ, standardized 
achievement test scores, and major course sequence. 
The Alumni Survey Questionnaire has three forms: (1) a 
mailed questionnaire for BOCES alumni; (2) a mailed question- 
naire for non-BOCES alumni; and (3) a telephone interview 
schedule used as a follow-up for nonrespondents. The first 
two forms of the Survey Questionnaire were identical except 
that the terms "high school" and "Vocational Center" were 
interchanged where appropriate. Three hundred twenty-one 
responses were gleaned from the mailed questionnaires and 
309 from the telephone interviews. The telephone question- 
naire was, in essence, an oral version of the mailed 
questionnaire: bojth the substance of the questions and the 
response formats were the same. 

Study findings are reported under several subheadings: 
cost analysis; school record profiles of BOCES and non-BOCES 
students; student performance compared with district ob- 
jectives; cost/benefit analysis; and cost/effectiveness 
analysis — BOCES vs. home school programs; curtailing ex- 
penditures while maintaining effectiveness; and increasing 
effectiveness without increasing total cost. The terms 
graduates, alumni, and students are used interchangeably 
throughout discussion of. the findings; for simplicity, the 
30 students who did not graduate are not differentiated 
from those who did. 

Among the findings the following are reported arid 
discussed : 

• There are considerable differences between students 
who select BOCES occupational programs (i.e., those • 
who spend some of their school time working in a 

job related to their training under a cooperative 
arrangement between the school and the employer) 
and students who select specializations in their 
home schools. Although most students in both 
groups viewed high school as the terminal point in 
their formal education, the non-BOCES group includes 
occupational majors such as business and home economics 
as well as nonoccupational majors such as art and 
foreign language. 

• The non-BOCES students scored significantly higher 
(.05 level) on IQ tests and achieved markedly higher 
percentile rankings in reading and mathematics 
achievement tests and in class rank. The differences 
are similar when males and females are analyzed 
separately, although they are more dramatic for 
males. 
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' • When the BOCES option became available, there was a 
decline in the ninth and tenth grade dropout rates 
but an increase in the eleventh grade d^opo^t "^e. 
Dropping out may have been deferred in. anticipation 
of the BOCES option but, when provided, the option 
may not have been sufficient to keep some students 
in school until graduation. 

• ror the entire period of the survey, 40% of the 
non-BOCES students indicated that they would not 
re-enroll in the same program if ♦'hey had to make 
^he chdice again, whereas 27% of ^he BOCES students 
would not re-enroll. 

• Of the cosmetology graduates who took- the licensing . 
examination, 94% passed; however, 38 out ot the 107 
graduates did not take the exam. 

• 82% of the male BOCES graduates ^1% of the male 
non-BOCES graduates located a full-time DOb within 
six months of graduation. 

• BOCES and non-BOCES alumni had similar character i^^^^^ 
in svich matters as conversation topics, leisure-time 
activities, and participation in organizations. 

• in the cost /benefit analysis, the only independent 

• bSces e?1ect. on hourly wages was that males earned 
more in their first jobs-an average difference of 
24«^. 

• The Trade Electricity program was the only program 
that had a benefit/cost ratio in excess of 1.0, 
after seven years it was 2.0 and growing. 

This study was conducted by a Professor of Education at 
the State University of New York at Buffalo for the Board 
of Cooperative Educational Services. 

EVALUATIVE COMMENTS: 

This is a complex study; various aspects of it , 
need to be assessed according to different criteria. The 
report is very thorough and well organized with a detailed 
table of contents and several tables facilitating use of the 
document. 



•^^For example, see Darcy, P erspective for a discussion 
of some of the pitfalls of cost-oenefit and cost-effective- 
ness analyses. 



The study is notable for at least two reasons: (1) The 
characteristics of respondents and non-respondents are com- 
pared; (2) graduates from two general types of programs are 
compared. It is perhaps unfortunate that no differentiation 
is made between the non-BOCES students enrolled in occupa- 
tional programs and those not enrolled in this area; 
however, the assumption presumably is based on the theory 
that all high school programs are in a^ sense "occupational" 
for students who do not intend to pursue further education 
or training. This is an issue that deserves -urth?r atten- 
tion. 



A Stu dy of the Status and Effectiveness of Cooperative 
Qffico Education in. New Jersey, 1968-69 . Carmela C. 
Kingston. Occupational Research Development Monograph 
No. 8. Trenton: The Research Coordinating Unit for 
Vocational-Technical Education, Bureau of Occupational 
•n Research Development, Division of Vocational Education, 
ilew Jersey State Department of Education, 1970, 40 pp. 
(ED 060 182) 

« 

DESCRIPTIVE ANNOTATION ' 

This study 'was done to determine the status of public 



Now Jersey and evaluate the effectiveness of those pro- 
grams in preparing students for office jobs. It was also 
meant to serve as a model for evaluation of other coopera- 
tive programs. It was hypothesized that COE students would 
be superior to noncooperative office education students 
in the following respects: that thoy would be employed 
sooner, would have more responsible positions, would receive 
higher earnings, would have greater job satisfaction, and 
would bo ratocJ more highly by their supervisors. 

The population comprised personnel from New Jersey 
secondary public schools having COE programs during the 
1968-69 school year: principals, business educ :ion de- 
partment chairpersons, cooperative education coordinators, 
cor: graduates and non-COE graduates (those who had been en- 
rolled in business classes and who had planned to secure 
o.fficG. jobs aff^r graduation but had not participated 
in coofH^rativo programs) ; employers of those graduates were 
also included. All the principals, chairpersons, and 
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secondary 




(COE) programs in 



coordinators and "as many as possible" of all th^ cooperative 
office education students and cooperating business firms 
were contacted. Further information regarding the sample 
is not given, but from the table "Percentage of Replies 
to Each Form" it can be determined that 735 of the 1,112 
employers contacted responded (66.1%) to part of the study, 
allhoigh fewer completed the employer "^ing form and that 
294 out of 400 (73.5%) of COE graduates and 476 out of 900 
(52.9%) of non-COE graduates responded to the Job Intorma- 
tion .Questionnaire. 

several instruments, with their development based on a 
literature review, were used to collect data: a question- 
naire concerning factual information, ^P^^i^^cAno? o^incl 
points sent to every New Jersey puhxic high ^chool princi 
pal; questions comprising background information, partici 
patim students, cooperating business^f irms, the COL 
coordinator, the State Department of Education, and addi- 
tional information sent to each COE coordinator; questions 
sent to graduates concerning their cooperative 30b exper- 
iences (if they had any) and their present ^obs.; 3ob- 
.erformance rating instruments sent to employers for com^J;^- 
Cion. The questionnaires were sent approximately four months 
after graduation. All instruments were in written form, 
and they incli.ied rating scales, multiple-choice questions, 
afd opin-ended questions. None of the instruments are 
appended nor- are reliability and validity discussed. 

Data were tabulated using ^BM computers; t- tests and 
chi-squares were used to test differences between COE and 

nnv rtr;,rJnates Several tables are provided. Among 
thr'fflnfings'a^d concLsiohs reported are the following: 

• 1)mployers were strong supp^i'ters of COE. 

• only 20% of COE students were able to participate 
iij school activities. 

• Approximately 80% of the COE graduates were employed 
full-time in an office. 

• COE graduates obtained jobs faster than non-COE 
graduates. 
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• There was no significant difference between COE and 
, non-COE graduate? at job entry in weekly gross 

wages received by office workers; however, at least 
one increase in pay was reported by significantly 
more COE graduates, and a significant difference in 
weekly gross vages was, evidenced five months after 
graduation. 

• Both COE and non-COE graduates experienced a high 
degree of job satisfaction. 

• Supervisors indicate that both groups need to 
demonstrate more initiative in job performance. 

• Although COE graduates received higher ratings than 
non-COE graduates in every area of job performance, 
the only ratings that were significantly higher 
(.05 or .01 levels) were in attitude toward work, 
attitude toward others, and overall ratings. 

Twenty-eight recommendations are made, ranging from use 
of continuous reporting and follow-up systems, to more 
varied job- training experiences, to improved school- 
placement functions. Suggestions for further research are 
made, including a study to determine v;hether or not more 
secretarial than clerical students are encouraged to 
participate in COE programs and a feasibility study to 
determine the desirability of establishing certification 
standards for coordinators. 

This study was done as partial fulfillment of the re- 
quirements for the author's doctoral degree. Permission to 
do the study was granted by the New uarsey Division of 
Vocational Education. 



EVALUATIVE COMMENTS: 

The most obvious problen with this study is the lack 
of sampling information. The reader is given no indication 
of poscrible rer.ponse bias or of why all the responses were 
not used m the analyses (although presumably this was 
the result of incomplete responses). Further, although the 
iiuthor states that instrument development was based on a 
litorahuro rev^iew, there is no indication of pilot tosttinn 
or of consideration cf reliability and validity. This i n- 
lormation.is necessary to determine how warranted the con- 
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For exploratory purposes and as a model for others, this 
study was quite thorough. Many factors were investi- 
gated from several perspectives, and statistical analyses 
were done. Further research along these lines should include 
consideration of intervening and control variables. 



* * * 



"The Success of Vocationally Trained Women in Traditionally 
Male Occupations." David J. Pucel. paper presented at 
the American Vocational Association Convention, New 
Orleans, Df»cembor 9, 1474, 20 pp. (ED 141 504> 



DESCRIPTIVE ANNOTATION; 

Wi*-h the passage of laws disallowing sex discrimination 
in employment practices, some educational institutions have 
stopped using sex as a basis for determining enrollment in 
particular programs. This study was done to examine a 
portion of the impact of this changing societal concept on 
vocational education. Four basic questions were asked: 

• Are women exercising their right to enroll in the 
traditionally male occupational training programs-*' 
of the Minnesota Area Vocational Technical Insti- 
tutes? 



• Do women who enroll in traditionally male programs 
judge the training programs similarly to men en- 
rolled in the same program? 

• Do women who graduate from traditionally male 
programs receive or perceive equal benefits in the 
world of work? 

• Do employers of the women who graduate from 
traditionally male programs judge them equal to men? 



a. traditionally male occupation was d^^ f./Ll... 
°S^cro^rupatL::s^w^rf id^n^ifiod tirol^ tL l970 Hinneso a 
Census Data. 



98 



Only 21 females who graduated in 1972 or 1973 from 
traditionally male occupational training programs and on 
whom the Minnesota Vpcational Follow-up system had data 
were identified. The study was continued in spite of this 
small sample because of the importance of the topic. The 
21 females were .drawn from eight program areas; 21 males 
were proportionately sampled from the. same areas to com- 
plete the study sample. Data obtained from the Minnesota 
Vocational Pollow-up System data tapes were analyzed using 
a chi-square technique*. Several percentage tables are pre- 
sented as well. 

Among the findings and conclusions discussed are the 
following : 

The largest female enrollment in traditionally male 
areas was 51% and 46% in chemical technology for 
1971-72 and 1972-73. 

• None^of the analyses of male and female judgments of 
their training programs yielded significantly 
different ♦*esults. 

• The first job salaries of the women were significantly 
lower than' those of the men. Pucel notes that data 
collected did not distinguish among the occupations 

in which the respondents were employed. 

• The employment-status differences one year after 
graduation were substantial, although not statisti- 
cally significant — 66.7% of the males were employed 
in training-related occupations compared to 38.1% of 
the females. 

• The women who received training equal to that of men 
were not 'as satisfied with the employment they 
entered as were the men. 

• None of the employer ratings of women were statisti- 
cally different from their ratinqs of men. However, 
the women were rated lower in terms of quality of 
work, quantity of work, job-related knowledge, and 
equlE^ment operation. Three possible reasons are 
offered: a difference in occupations, bias against 
women, and an actual difference in performance. 

This study was conducted by a Professor of Vocational 
and Technical Education at the University of Minnesota 
who is also Director of the Minnesota Vocational Follow-up 
System. 
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EVALUATIVE COMMENTS: 

The author cautions against making weighty decisions on 
the basis of this study because the sample was so small. 
Still, this has value as an exploratory study because it 
raises questions tha^ can be investigated through further 
research. ' . » 



* * * 



Vocational Education Planning Districts in Ohio; An 

Economic Evaluation of Foregone Benefits from Limited 
Participation . I . A. Ghazalah. Columbus: Division of 
Vocational Education, Department of Education, State of 
Ohio, February 1975, 56 pp» 

; 

DESCRIPTIVE ANNOTATION: 

The author states, "The purpose of the economic analysis 
of publicly financed programs is to assist decision- 
makers in. the allocation of a given set of scarce resources 
among competing uses." This particular stuc3^ was under- 
taken because "knowledge about the potential net benefits ... 
to society from increasing the percentage of senior high 
school students who receive job training should assist in 
the appropriate development of vocational education in 
[Ohio]." The specific objectives of- the study were as .. 
tollows: ' ^ 

• To estimate, on the basis of enrollment during the 
fiscal year 1973,, Total Net Social Benefits (TNSB) 
from vocational programs at 11th and 12th grade levels 
in each of four sample vocational education planning 
districts 

0 To estimate the TNSB that would accrue from increasing 
enrollment in each sample 'to 40% of the average 
daily membership in those grades 

• Based on saiaple data, to estimate for -ach planning 
district the TNSB for the fiscal year 197 3 and the 
potential, increase from increased enrollment ' 

• To provide absolute, as well as, relative economic 
value of the array of vocational education programs 
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A sample of foui* vocational planning districts was 
solected for the study: (1) a single vocational district 
with eleven member schools; (2) a »nultiple vocational 
district with five member schools; (3) a small joint voca- 
tional school district with six member schools; and (4) a 
large joint vocational school district with sixteen member 
schools. 

Three sources of data were used in the study: schools . 
and school districts; the state Division of Vocational 
Education; and published data from the U.S. Department of - 
Labor 'and the Ohio Department of Education. Interviews were 
held with school superintendents and. administrators to 
obtain and clarify financial, enrollinent , and program data 
about each school and follow-up data on its students. 
Trainees in each school completed questionnaires concerning 
demographic and earnings (from part-time employment during 
training) variables. The Division of Vocational Education 
supplied unpublished enrollment data. The published data 
sources provided wage rates, labor- force participation 
rates and survival probabilities, and cost- per- pupil data'.' 

Although the objectives of vocational education are 
multi(Ume':sional, the state-of-the-art in economic analysis 
does not allow specification of a single functional rela- 
tionship that uniquely encompasses all the dimensions^ 
Therefore, "Since vocational education is principally aimed 
at increasing the productivity of human resources, this 
\.study [is limited] to an evaluation of vocational education 
as an investment goal, i.e., in terms of- its contribution 
to the objective of raising the output of goods and services 
in the economy as measured by the increase in the earning 
power of participants in vocational education." Costs and 
benefits to society as a whole are evaluated. In the 
analyses, "Social Present Value I" is calculated with the 
asisumption that without vocational education the student 
would have dropped out of school after the 10th grade to 
enter the labor market. "Social Present Value II" is cal- 
culated with the assumption that without vocational educa- 
tion, the student would have completed the 11th and 12th 
qrades in the academic curriculum. Separate calculations 
were made for males and females. 

Based on his calculations, the author concludes that, 
on in absolute basis, all the evaluated vocational 
programs — except child care — have positive social present 
values, making them worthwhile investments. The relative 
value varies with program, with methods of calculation of 
social present value, and with sex. Tables detailing 
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these findings are provided. The computed estimates in- 
dicate that increasing vocational education partiv:ipation 
by 40% would result in a statewide increase in net social 
benefits from $108;918,52e to $326,951,424. 

The author of this document is a Professor of Economics 
at Ohio University. The project was supported by funds 
from the Ohio Department of Education, Division of Vocational 
Education. 

EVALUATIVE COMMENTS: 

The assumptions underlying the economic anlysis and 
the equations used are discussed in detail. However, the 
data used in the analysis are not. Sources of data are 
listed but there is no indication of how reliable the dat^a 
sources are or how valid the trainee questionnaire was, 
and no indication of how any possible limitations in the 
data would affect the results. 

« 

* * * 



Vocational Technical and Adult Education: Student Follow - 
« up Study of 1974"1975 Completions . Jim Preston. Sara- 
sota County, Florida: Sarasota county Board of Public 
Instruction, May 21, 1976, 68 pp. (ED 124 SZS) 



DESCRIPTIVE ANNOTATION: 

t. 

This is a report of the fourth annual study of Voca- 
tional Technical and Adulfc Education (VTAE) conducted in 
Sarasota County, Florida. It is based on data analysis pro- 
vided by the local District Data Processing Department 
before the surveys were sent to the state centralized re- 
porting station. Thus, this report was developed to give 
timely data to decision- makers before the narrative de- 
scriptions were written. The survey questionnaire addressed 
employment availability, employment status, job classifi- 
cation, quality of training, reason for emplov».ient outside 
a training-related area, average weekly pay, source of 
assistance in finding job, and postsecondary education. 
It was hoped that the information provided would increase 
the ability of decision-makers to allqcate resources and to 
assist in planning, reviewing, and evaluating the vocation- 
al proqrams in Sarasota County. 
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This survey of 1974-75 completers war conducted in 1976. 
An alert card was mailed to the sample in January and the 
second survey mailing and personal contacts to non- 
respondents were made in February. 

Tha survey population was 1025 Vocational Preparatory 
program completers from the Sarasota County Vocational- ' 
Technical Center and three high school Home Economics and 
Cooperative programs. (A listing of specific courses and 
respective response rates is provided.) Many hours were 
spent in deleting from the state computer listings of voca- 
tional students the names of those taking only one course 
and that course for some reason other than job preparation. 
Names of 430 were deleted before the questionnaires were 
mailed. The response rate was 63%, which was raised to 67% 
by additional responses received after the deadline. Though 
not included in this report, these late responses w^e ana- 
lyzed with very similar results, 

A pilot instrument for the annual studies had been * 
deyeloped in, 1971 and changed little until the time of 
this survey. Changes were made to accommodate federal re- 
quirements for collecting veteran and Comprehensive Em--' 
ployment and Training Act (CETA) data. The instrument 
appears in the report. The author states lhat, filthough 
validity studies per se have not been done, the high re- 
sponse rate and the lack of differences in responses be- 
tween prompt and late responders should increase confidence 
m the data. 

Tabular percentage data for- the questions addressed are 
provided for each of the following groups: (1) the total 
survey sample, (2) students enrolled in home economics and 
cooperative programs, (3) total enrollment at the Vocational 
Technical Center (VTC) , (4) VTC students enrolled in health 
occupat.icns, industrial programs, and office occupations. 
(5) secondary students, and (6) postsecondary students. 

Amorg the findings reported are (1) that of those avail 
for employment in a training-related area? Sofwe^e lm- ^^^^^ 
ployed— 64% of them in their area of training or a related 

and (2) that, after completing their secondary pro- 
grams, 2^% of the respondents went on to post-secondary 
training— about half of them in a related program. 

*rc-'^^^,^""'^^^ studies are sponsored by State Department 
officials who supply survey forms and set the schedule for 
mailing them; this report was prepared by Jim Preston, Pro- 
gram Specialist, Placement and Follow-up, Sarasota County 
Board of Public Instruction, 
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EVALUATIVE COMMENTS i 

This report consists, for the most part, of sununary 
data tables, i.e., frequency (percentage) charts, with very 
little interpretation of results. A benefit of this format 
is that the report can be produced quickly, giving decision 
makers access to data before decisions have to be made. 
Another ,adv.antage is that specific information sought often 
can be more easily found in a table than in narrative form. 
However, there are disadvantages to this format as well. 
Frequency tables can be misleading in that they present only 
frequencies of occurrences— ^information that may be crucial 
to the interpretation of differences in frequencies is 
lacking. For this reason, those who consider the "data 
without interpretation" format should consider the disad- 
varvtages as well as the advantages. 

* * * 

* * * 
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IV. EPILOGUE: RECAPITU^^ATION AND ASSESSMENT 

This epilogi^ summarizes substantive findings reported 
in the descriptive annotations and methodological issues 
discussed in the evaluative comments. It is intended to give 
the reader a quick overview of data-based claims made about 
sc«ne outcomes of, vocational education. Readers are cautioned 
that findings presented here are inferences basad on a review 
of the thirty-one studies included in this bibliography. 

Several of the studies were designed specifically to 
provide information for describing o}p foy improving particu- 
lar programs, rather than to add to a body of knowledge about 
vocational education outcomes. Still, these studies are 
easily accessible to a wide audience, and their results may 
mistakenly be regarded as widely generalizable by individuals 
who are unfamiliar with the canons of research. It is hoped 
that this epilogue will be helpful in indicating where fur- 
ther research is needed and where improvements can be made 
in conducting and reporting evaluation studies Thus, it is 
meant to be helpful both to readers of evaluation reports 
and to evaluators. 

••< 

The methodological issued discussed here in reference 
to the annotated studies are discussed more generally and 
in greater detail in Section I I above and in the references 
provided at the end of that section. 

It is difficult to draw conclusions based on this set 
of studies because of several definitional problems: 

• The populations studied are often not ■ 
clearly defined. 

• When populations are defined, reasons under- 
lying the choice of definitions are not stated. 

• Reasons underlying the choice of operational 
definitions-'^ 'Of the outcomes under study are 
not stated. 



^^An operational dWinitior indicates how an outcome is 
to be measured. For exaViple, intelligence is frequently 
operationally defined by scores on a standardized intelli- 
gence test. Job satisfaction can be operationally defined 
by scores on a job satisfaction rating scale or by an 
employee's tardiness or use of sick lea^e. 
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After a discussion of difficulties created by definitional and 
other problems, the epilogue turns to methodological issues as 
they apply to several types of outcomes: 

# Employment and education status 
• • Earnings 

# Student satisfaction with training 

• Job satisfaction 

• Job performance 



Populations 

In general, the populations for these studies are sets 
of individuals who have been enrolled in vocational programs. 
However, terms ^re not used consistently in the different 
studies. For example, "high school vocational gr'a'duates" as 
used in one study may indicate those graduates who had taken 
a sequence of vocational courses in either , 'the eleventh or 
twelfth grade, whereas in another study, the term may indi- 
cate graduates who completed two credits in. a vocational 
program during a school year. 

Not only is there inconsistency among studies in the use 
of various terms related to £he population, but often the 
terms used are not clearly defined. This makes it difficult 
to determine when comparability exists. For example, studies 
may include "early leavers with marketable skills" along with 
fJrogram completers without specifying how such early leavers 
arc distinguished from dropouts. 

Presumably there are reasons underlying the specific 
definitional choices made. If those reasons were stated, it 
might be possible to determine what populations, although 
different in specifics, are comparable enough to be considered 
together by those wishing to draw conclusions. The thirty-one 
studies reviewed here are comparable, with respect to their 
•populations, in that all address those students, graduates, 
or leavers characterized as "vocational" by the authors; 
however, it is difficult to determine if the populations are 
comparable with respect to any finer discriminauions. 



Operation al Ho finitions of Outcomes 

m most of the studies the operational definitions of 
the outcomes investigated are implicit, if not explicit. 
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For example, the reader can discover how an outcome was de- 
fined by perusing the questionnaire used in the study. In 
some cases, however, studies report on an outcome such as 
employment in a training-related field without indicating how 
"trainin,g-related field" is defined or determined and without 
appending the instrument. Thus, identical or similar labels 
may be used in different studies for outcomes that are not 
comparable. Table IV provides a list of several outcomes de- 
fined in ways that prevent comparisons among studies, along 
with examples of the questions left unanswered by definitions 
that are not explicit. 



Table IV 

OUTCOMI-IS OFTEN NOT CLEARLY DEFINED— 
AND RESULTANT UNANSWERED QUESTIONS ' 

• Employment in occupational program area/in area related to 
occupational program (How arc the two differentiated?) 

• Job sat i3 faction (rtow is it evidenced? Is it an all-incXu- 
. rfivo term ofc, subdivided into areas of satisfaction such as 

earnings and working conditions?) 

• Satisfaction with aducation or training (How is it deter-" 
mined? Did training help students find or keep. a job? 
W<iuld students recommend program to others?) 

• Solf-imag-e» self-concept » self-estoem (As used* are 
those all labels for the same construct? How is the 
feeling measured?) 

• Earning s (Does the term refer tp hourly earnings 

or average annual income? How can various earnings 
variables be compared?) 

• Emp loymen t/Unemp loy men t rates ;How are they calculated? 
Is the employment rate based only on those available 
for work?) 



Other General Issue s 

The non-comparability of both populations and outcome 
definitions makes any attempt to draw conclusions from this 
sot of studies difficult at tne outset • For this reason^ some 
other issues related to comparability of studies are relatively 
loss important: for example^ differences in response rates 
and data collection and analysis procedures. Stilly. these 
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and other methodological issues are relevant to the asset s- 
rr.ent of outcomes in any given study and are not adequately 
addressed in several of these studies. 

One such issue is response bias, which is particularly 
likely to occur in cases where the response rate is small. 
Even the studies that do attempt to analyze response Mas fail 
to acknowledge that the factoid (s) influencing tendency to re- 
spond may interact with the outcome (s) being assessed. When 
this is the case, bias cannot be analysed easily. For exam- 
ple, if the unemployed tend not to respond to employment status 
questionnaires, this bias may not b? discovered thrpugh anal- 
ysis of demograpliic variables associated with the respondents 
and those nonrespondents who can be reached by 'phone. 

Another issue is the reliability and validity o*f the data- 
collection tools or techniques. F 5w studies discuss the degree 
to which valid and reliable data have been collected. For ex- 
ample, when relatives are asked to report on an "unreachable" 
individual's employment status, there should be indication of 
whether the accuracy of the report was determined. Reliabil- 
ity and validity should be assessed even when the data collect- 
ion appears to be straightforward. 

A list of questions related to these and other issues is 
provided in Table V. These questions are left unanswered in 
most of the reports of studies annotated in. this bibliography. 
A discussion of methodological issues related to particular 
outcomes follQWs the table. 



Table V 

METHODOLOGICAL QUESTIONS UNANSWERED 
IN MANY OUTCOME STUDIES 

# Wore quo<5ttons asked so Icng after the event (e.g., vocation- 
al trainiri'j, first job) that one might question accuracy of 
recall? 

# Wa.s thf» study sample reprfiStnitat i ve of the* stated population? 
Wero appropriate samplinq procedures used? Was the response 
rafe sufficiontly largo? Was response bias analyzed'^ 

# Wei*: th»» metnodoloqy powerful enouqh (sonsitivo instruments, 
appropiiat.o data analysis, fuif f icient ly larqe samf^le si^'.e 
etc.). to detect effects of the vocational program? 
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Table V (Continued) 

METHODOLOGICAL QUESTIONS UNANSWERED 
IN MANY OUTCOME STUDIES 

Did the research design covs*r the probable threats to valid- 
ity? What assurance is there that the observed result was a 
function of vocational education rather than of some unmea- 
sured variable (s)? What comparisons were made? To what 
deqree can the results be generalized to vocational progz*ams 
in other settings? 

Were data collected from reliable sources? Was any 
verification attempted? 

Was the instrumentation or dr.ta collection tool/technique 
carefully developed, pilot-tested, and revised to promote 
reliability and validity? 

Were data collection conditions optimal—questionnaires 
easy to complete, trained interviewers, comparable con-* 
ditions of administration? 

Are results that are statistically significant also 
meaningful? How much variance in the effect is asso- 
ciated with vocational education? 

Can differences confidently be attributed to vocational 
education rather than to chance? Were data collected in 
a way that permits statistical analysis? Was statistical 
analysis done when appropriate? 

When data were statistically analyzed, were appropriate 
statistics used? What assumptions were violated and with 
what probable effects? 

To what dpgroe might factors/tendencies such as social 
dostrabi li ty, acquiescence, and nalo-effect influence 
quei',tionnai ro responses: And to what degree might some' 
of those factors affect tendency to respond? 

When rating scales wore used, what was the rationale 
for choosing rosponso alternatives and assigning values 
to them? Might the structure of the altornativos have 
induced a positive or nuqative response bias? What was 
the rationale for collapsing across alternatives when 
tht^ responses wore analy/,od? 
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Employment and Education Status 

Some researchers differentiate "employment in t^rea of 
training" from "employment in traininq-related ureas," and 
others combine the two. Few describe the criteria for deter- 
mining whether a job is actually in the area of training or 
is related to training, and if so, to what degree. Somc^ ask 
the respondents whether their jobs were training- related 
whereas others ask for job titles and use the Dictionary o r 
Occupational Titles to make a determination. Some report 
percentages of graduates or leavers with full-time jobs or.ly, 
some report these with part-time jobs as a separate category, 
and others do not indicate their criteria for placing an 
individual in an employment category. Although the studies 
generally report graduates' employment status soon after the 
students left school (e.g., from six months to one year) the 
data may have been collected from six months to eight years 
later (does a job held eight years ago seem to have the same 
degree of training-relatedness to the respondent es it did 
during 'the first six months after graduation?) . 

Some studies report the percentage of employed graduate^ 
based on the total number of .survey respondents whereat 
others use the number available for employment as the base. 
Some report employment status by occupational program area, 
some by sex, and others provide no partitions. F^w studies 
compare employment status of vocational students to any other 
set of students or to any standards at all.-'^ 



•^^Reporting rates of employment rather than unemployment 

creates problems of calculation and comparison. People 
who are employed ("worked at all as paid employees" during 
V tho survey week, according to U. S. Bureau of Labor Statis- 

tics [BLS] definitions) are easy enough to count, but in cal- 
culating a rate of employment, one must define and gather 
vdata for an appropriate base. This may be all labor force 
participants, only those people who are 'available for emplov- 
ment," or some other population set. With the base determined, 
however, what comparisons can he made to give meaning to the 
rate calculated for the vocational graduates? Labor force 
statistics published by state and tederal government agen- 
cies focus on unemployment (i.e., number of unen.ployed workers 
divided by the civilian labor force), rather than employment. 
Unless the base used in computing employment rates for voca- 
tional students is the same as the BLS base used in calculat- 
ing unemployment rates, it is not possible to convert the 
rat€3s for purposes of comparison. Thus, if an evaluation 
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This variation makes it difficult, if not impossible, to 
draw valid conclusions i It does appear that on the average, 
a greater percentage of postsecondary vocational graduates 
find employment and particularly training-related employment 
than, do high school vocational graduates. The reasons, how- 
ever, are not addressed; among the possibilities are that 
high school graduates prefer to 'seek further training or 
receive more specialized training and are therefore more mar- 
ketable, and that employers consider young adults more desir- 
able than teenagers as employees. It also appears that gradu- 
ates of some occupational program areas are more likely than 
graduates of others to obtain employment and to obtain train- 
ing-related employment. Further,, the occupational program 
areas may interact with level of education. 

Some areas for further research are obvious: definitions 
must bo specified; comparisons need to be made; the correlates 
of employment/unemployment status (for example, occupational' 
program, sex, educational level) need to be investigated. 

The findings relevant to educational status suffer from 
the same general "problems. It appears that more high school 
graduates seek further education or training than postsecondary 
graduates, certainly not an unexpected finding. 



Earnings 

Conclusions about earnings are even more difficult to 
draw than conclusions regarding employment or education status. 
When results are reported strictly as dollar amounts, any 
comparison between studies failing to consider the year of the 
study and geographic differences in prevailing wage rates 
would not be meaningful. Not only do many studies report 
dollar amouijts without providing other data necessary for 
interpretation, but they also fail to compare earnings of 
vocational graduates with those of other graduates or with 
other standards, and they fail to partition graduates by pro- 
gram area. Further, some studies report hourly earnings, 
others report beginning weekly gross earnings, and others 
•'accumulated income." In order for earnings to be an 



report indicates an employment rate of 92% for voca- 
tional graduates whereas the official unemployment rate for 
the relevant age group is 16%, anyone reporting that the 
unemployment rate of vocational graduates was only half that 
of the general population would be in error, and any infer- 
ences drawn from that cor elusion would be misleading. 
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appropriate outcome criterion for evaluating vocational edu- 
cation, the research community must agree on some appropriate 
comparison groups and on some standard, metrics. ^ 



Student Satisfaction with Training 

Students are considered highly satisfied with training 
when they agree with statements like the following: 

• If I had the choice to make again, I would choose 
the same program. 

• • I would recommend my program to other students. 

• My training helped me to get my job. 

• My training has helped me on the job. 
. • I v;as satisfied with my training. 



This outcome was assessed through written or telephone ques- 
tionnaires, sometimes with multiple items, sometimes only one, 
sometimes using a yes/no format, sometimes using a five-point 
rating scale. Although some researchers indicate that they 
pilot-tested their instruments, few discuss any aspect of. 
instrument development or validation. Without' careful devel- 
opment and testing there is no indication to what degree re- 
spondents' answers are influenced by response sets such as 
•acquiescence, halo effect, or leniency by ambiguous or un- 
clear statements; or by inadequate response choices' (i.e., no 
response fitting what the respondent wants to indicate) . Thus 
the only conclusion easily drawn is that, when asked, 'most 
graduates will indicate a high degree of satisfaction with 
their training. 

Further research is needed in two areas: instrumentation 
and correlates of satisfaction. If student satisfaction with 
training is important, it is certainly important to know what 
fosters and what hinders it; it may also be important to know 



^^Acquiescence is the tendenc^y to make pcoitive responses 
halo effect refers to the tendency to say good things about 
every characteristic based, for example, on a general positive 
feeling about what or who is being rated, or on one strong 
point; and leniency is the tendency to give the more positi'.e 
rating when unsure of which rating to give. 
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what effects it has^ To adequately assess any aspect of train- 
ing satisf actxon, the instruments must be carefully developed 
and their validity and reliability must be analyzed. Much 
research has gone into the development of instruments for 
assessing various attitudes; such research is needed in the 
area of training satisfaction as well. 



Job Satisfaction 

Job satisfaction has been determined in two ways: through 
use of a published job. satisfaction scale and by asking a single 
question on a questionnaire or telephone interview, something 
like, "Are you satisfied with your job?" The issues in assess- 
ing job satisfaction, then, are similar to those in measuring 
training satisfaction, and the suggestions for further research 
are analogous. Further, some studies suggest that job satis- 
faction is higher for graduates in training-related jobs, but 
others do not distinguish those with training-related jobs 
from those with other jobs. This, then, is an area needing 
further research. 

There is a body of literature on job satisfaction, indi- 
cating that it is a multidimensional concept amenable to assess- 
ment from various perspectives. For example,, job satisfaction 
may be inferred, in part, from turnover rates and use of sick 
leave. Vocational education researchers would do well. to ex- 
amine the literature and to do further research on what kinds 
of. measurement are most, appropriate for evaluating vocational 
education against a job-satisfaction criterion. 



Job Performance (Attainment of Occupational Competencies ) 

One way to evaluate job performance is to ask employers 
how their employees perform on the job, the only technique 



"^^See, for instance, Robert P. Quinn and Martha S. Baldi 
Uo Mandi lovitch. Education and Job Satisfaction: A Question- 
ablo Payoff (Washington, D.C. : U.S. Department of Health, 
Education, and Welfare, National Institute of Education, - March 
1077) . 

^'^Soe Kenney E. Gray, Marie Abram, and Floyd McKinnoy, 
Vocational Education Measures: Instruments to Survey Former 
Stuck^nts and Their Employers (Columbus; The National Center 
for Research in. Vocational Education, The Ohio State Univer- 
sity, 1978) for a review of some of -the instruments available. 
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*( beyond asking the graduates whether their training contributed 
to, their job performance) used in any of the thirty-one studies 
presented in- this bibliography. This is not to .say that per- 
formance testing, is not done*, but it does not appear to be 
done often in vocational education outcomes evaluation. 

Only a few of the thirty-one studies reviewed in this 
bibliography sought employer perceptions, and those that did 
are fraught with problems limiting the degree to whibh their 
results' contribute to knowledge about the job performance of 
vocational education graduates. One problem is the use of 
presumably untested, unvalidated rating scales. As discussed 
in the section on training satisfaction, without careful de- 
velppment and testing there is no oindicatfoh to what degree 
respondents' answers are influenced by factors unrelated to 
tho substantive content of the rating scale. 

» 

Another problem is that data indicating the extent of 
employers' experience with vocational graduates have not been 
collected (for example, tota^^ number hired, percentage of 
employees from Vocational progr:;ms, how long vocationally- 
trained employees have been used) . Further, the data that 
have been collected have not been verified. For example, 
hiring-prcf erence data consis'ting of questionnaire responses 
have not been verified by inspection of personnel records. 

Finafly, comparative data have not been collected. Some 
questionnaire items asked whether graduates of vocational pro- 
grams were better ' employees than graduates of general programs, 
but r ne compared' performance ratings of vocational and non- 
vocational graduates. 

With all these difficulties, th^ only conclusion that 
can be drawn from these • studies is that employers tend not to 
express dissatisfaction with employees who are graduates of 
vocational programs. Further research is needed to overcome 
tho problems discussed above. More studies should address 
multiple factors of performance to discover whfether vocational 
education affects some factors but not others. It cannot be 
overemphasized that there are several problems inherent in the 
use of rating scales that can be summarized by the statement 
that a rating scale often reveals more abou^t the rater than 
about the substantive content of the scale. Thus, it is very 
important that rating scales be carefully derives! and* 
thoroughly tested and that verification be done j^here possi- 
ble through use of other assessment techniques.^"' 



■ r . 

'^'^See Darcy, Perspective , for a discussion of types of 
data that can be collected for evaluation purposes. 
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other Outcomes 



Other outcomes addressed in some of these studies include 
leadership, voting behavior (as a proxy for societal partici- 
pation) , self-image, academic standing, success of women in 
traditionally male jobs, and dropout rates. . These outcomes . 
were not addressed by enough studies to permit synthesis of 
findings, x In most cases, the research suffered from the • 
limitations discussed above in reference to particular out-' 
comes . 
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